2024-09-20T23:28:23.886 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-20T23:28:23.904 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-20T23:28:23.987 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089 branch: squid description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7913089' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_REFRESH_FAILED log-only-match: - CEPHADM_ sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 9344 sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 sleep_before_teardown: 0 subset: 56/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGRRp0xZ9xalleg0pVUDxKn9BXZfA9QEJBccHLUWnqsNFCbhkH83YArTsqqCzTglBYQI6lzNiBRn1TjQDnFkdU0= smithi144.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP39rIbzw4SgEx4Me3SKkeTIZYTe0wJ3ibna+Xg3Umg2ncOoRPR5CMfRiqrghqem38Do5rpG/p/mOipzZbrzPks= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-09-20T23:28:23.987 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-20T23:28:23.988 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-20T23:28:23.988 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-20T23:28:23.990 INFO:teuthology.task.internal:Checking packages... 2024-09-20T23:28:24.011 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9e2ac70bb7aea67d34f0d50002344abbe7803736' 2024-09-20T23:28:24.011 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-20T23:28:24.011 INFO:teuthology.packaging:ref: None 2024-09-20T23:28:24.011 INFO:teuthology.packaging:tag: None 2024-09-20T23:28:24.011 INFO:teuthology.packaging:branch: squid 2024-09-20T23:28:24.011 INFO:teuthology.packaging:sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:28:24.011 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-20T23:28:24.247 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-20T23:28:24.249 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-20T23:28:24.255 INFO:teuthology.task.internal:no buildpackages task found 2024-09-20T23:28:24.255 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-20T23:28:24.414 INFO:teuthology.task.internal:Saving configuration 2024-09-20T23:28:24.429 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-20T23:28:24.500 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-20T23:28:24.525 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 23:23:35.334640', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGRRp0xZ9xalleg0pVUDxKn9BXZfA9QEJBccHLUWnqsNFCbhkH83YArTsqqCzTglBYQI6lzNiBRn1TjQDnFkdU0='} 2024-09-20T23:28:24.544 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi144.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-20 23:23:35.332901', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBP39rIbzw4SgEx4Me3SKkeTIZYTe0wJ3ibna+Xg3Umg2ncOoRPR5CMfRiqrghqem38Do5rpG/p/mOipzZbrzPks='} 2024-09-20T23:28:24.544 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-20T23:28:24.612 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-09-20T23:28:24.612 INFO:teuthology.task.internal:roles: ubuntu@smithi144.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-09-20T23:28:24.612 INFO:teuthology.run_tasks:Running task console_log... 2024-09-20T23:28:24.811 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fcca2130160>, signals=[15]) 2024-09-20T23:28:24.812 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-20T23:28:24.929 INFO:teuthology.task.internal:Opening connections... 2024-09-20T23:28:24.929 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:28:24.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:28:25.007 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi144.front.sepia.ceph.com 2024-09-20T23:28:25.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:28:25.085 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-20T23:28:25.137 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-20T23:28:25.156 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-20T23:28:25.156 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T23:28:25.214 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T23:28:25.215 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T23:28:25.216 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-20T23:28:25.237 DEBUG:teuthology.orchestra.run.smithi144:> uname -m 2024-09-20T23:28:25.255 INFO:teuthology.orchestra.run.smithi144.stdout:x86_64 2024-09-20T23:28:25.255 DEBUG:teuthology.orchestra.run.smithi144:> cat /etc/os-release 2024-09-20T23:28:25.313 INFO:teuthology.orchestra.run.smithi144.stdout:NAME="CentOS Stream" 2024-09-20T23:28:25.313 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION="9" 2024-09-20T23:28:25.313 INFO:teuthology.orchestra.run.smithi144.stdout:ID="centos" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:ID_LIKE="rhel fedora" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:VERSION_ID="9" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:PLATFORM_ID="platform:el9" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:ANSI_COLOR="0;31" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:LOGO="fedora-logo-icon" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:HOME_URL="https://centos.org/" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-20T23:28:25.314 INFO:teuthology.orchestra.run.smithi144.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-20T23:28:25.315 INFO:teuthology.lock.ops:Updating smithi144.front.sepia.ceph.com on lock server 2024-09-20T23:28:25.337 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-20T23:28:25.344 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-20T23:28:25.470 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-20T23:28:25.471 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-20T23:28:25.473 DEBUG:teuthology.orchestra.run.smithi144:> test '!' -e /home/ubuntu/cephtest 2024-09-20T23:28:25.490 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-20T23:28:25.699 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-20T23:28:25.699 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-20T23:28:25.701 DEBUG:teuthology.orchestra.run.smithi144:> test -z $(ls -A /var/lib/ceph) 2024-09-20T23:28:25.810 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-20T23:28:25.952 INFO:teuthology.run_tasks:Running task kernel... 2024-09-20T23:28:25.984 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-20T23:28:25.984 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-20T23:28:25.985 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T23:28:25.985 DEBUG:teuthology.orchestra.run.smithi144:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-20T23:28:26.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:28:26.003 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-20T23:28:26.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:28:26.004 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-20T23:28:26.060 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-503.el9.x86_64 2024-09-20T23:28:26.060 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-503.el9.x86_64 2024-09-20T23:28:26.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-09-20T23:28:26.062 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-503.el9.x86_64 2024-09-20T23:28:26.062 INFO:teuthology.task.kernel:Running kernel on smithi144: 5.14.0-503.el9.x86_64 2024-09-20T23:28:26.062 DEBUG:teuthology.orchestra.run.smithi144:> sudo yum install -y kernel 2024-09-20T23:28:28.277 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.3 MB 00:00 2024-09-20T23:28:28.562 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.3 MB 00:00 2024-09-20T23:28:33.061 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 5.5 MB/s | 20 MB 00:03 2024-09-20T23:28:35.863 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - AppStream 3.3 MB/s | 20 MB 00:06 2024-09-20T23:28:37.412 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-20T23:28:38.879 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 19 kB 00:00 2024-09-20T23:28:39.675 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-20T23:28:40.188 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-20T23:28:41.972 INFO:teuthology.orchestra.run.smithi144.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 19 kB 00:00 2024-09-20T23:28:42.769 INFO:teuthology.orchestra.run.smithi144.stdout:Extra Packages for Enterprise Linux 34 MB/s | 23 MB 00:00 2024-09-20T23:28:45.155 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-09-20T23:28:47.003 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T23:28:47.003 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T23:28:47.082 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-20T23:28:47.088 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-20T23:28:47.089 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-20T23:28:47.089 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:28:47.089 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 79 M 2024-09-20T23:28:47.089 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 126 M 2024-09-20T23:28:47.089 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-20T23:28:47.774 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 64 kB/s | 36 kB 00:00 2024-09-20T23:28:48.290 INFO:teuthology.orchestra.run.smithi144.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-09-20T23:28:48.940 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 9.1 MB/s | 16 MB 00:01 2024-09-20T23:28:50.107 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 12 MB/s | 35 MB 00:02 2024-09-20T23:28:50.180 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-20T23:28:50.180 INFO:teuthology.orchestra.run.smithi144.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-20T23:28:50.268 INFO:teuthology.orchestra.run.smithi144.stdout:Dependencies resolved. 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: Package Architecture Version Repository Size 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:Installing: 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:Installing dependencies: 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:28:50.273 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction Summary 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout:================================================================================ 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout:Install 4 Packages 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout:Total download size: 79 M 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout:Installed size: 126 M 2024-09-20T23:28:50.274 INFO:teuthology.orchestra.run.smithi144.stdout:Downloading Packages: 2024-09-20T23:28:50.414 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 11 MB/s | 29 MB 00:02 2024-09-20T23:28:50.414 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-20T23:28:50.414 INFO:teuthology.orchestra.run.smithi137.stdout:Total 24 MB/s | 79 MB 00:03 2024-09-20T23:28:50.658 INFO:teuthology.orchestra.run.smithi144.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 342 kB/s | 36 kB 00:00 2024-09-20T23:28:51.098 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-20T23:28:51.250 INFO:teuthology.orchestra.run.smithi144.stdout:(2/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 50 MB/s | 35 MB 00:00 2024-09-20T23:28:51.460 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-20T23:28:51.461 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-20T23:28:51.517 INFO:teuthology.orchestra.run.smithi144.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 33 MB/s | 29 MB 00:00 2024-09-20T23:28:51.717 INFO:teuthology.orchestra.run.smithi144.stdout:(4/4): kernel-core-5.14.0-508.el9.x86_64.rpm 13 MB/s | 16 MB 00:01 2024-09-20T23:28:51.717 INFO:teuthology.orchestra.run.smithi144.stdout:-------------------------------------------------------------------------------- 2024-09-20T23:28:51.717 INFO:teuthology.orchestra.run.smithi144.stdout:Total 55 MB/s | 79 MB 00:01 2024-09-20T23:28:52.403 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction check 2024-09-20T23:28:52.430 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-20T23:28:52.431 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-20T23:28:52.774 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction check succeeded. 2024-09-20T23:28:52.775 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction test 2024-09-20T23:28:53.733 INFO:teuthology.orchestra.run.smithi144.stdout:Transaction test succeeded. 2024-09-20T23:28:53.733 INFO:teuthology.orchestra.run.smithi144.stdout:Running transaction 2024-09-20T23:28:54.884 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-20T23:28:55.544 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T23:28:55.633 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:28:56.150 INFO:teuthology.orchestra.run.smithi144.stdout: Preparing : 1/1 2024-09-20T23:28:56.796 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-20T23:28:56.902 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:28:58.393 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:28:58.543 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:28:59.639 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:28:59.763 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:29:03.406 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:29:03.451 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:04.635 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:29:04.671 INFO:teuthology.orchestra.run.smithi144.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:08.283 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:09.563 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:46.521 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:46.561 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:46.845 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:46.846 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T23:29:46.846 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:29:46.846 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T23:29:47.723 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T23:29:47.724 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:29:47.724 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-20T23:29:48.084 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T23:29:48.164 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:48.164 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T23:29:48.164 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T23:29:48.164 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T23:29:48.164 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T23:29:48.164 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-09-20T23:29:48.164 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T23:29:48.165 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T23:29:48.186 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:48.222 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:48.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T23:29:48.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-09-20T23:29:48.229 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:48.255 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-09-20T23:29:48.255 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-09-20T23:29:48.255 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-09-20T23:29:48.255 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 82.el9 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8274670 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-09-20T23:29:48.256 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:29:48.257 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T23:29:48.258 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T23:29:48.258 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-508.el9.x86_64 2024-09-20T23:29:48.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T23:29:48.629 INFO:teuthology.orchestra.run.smithi144.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:48.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-20T23:29:48.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-20T23:29:48.629 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-20T23:29:48.913 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-20T23:29:49.510 INFO:teuthology.orchestra.run.smithi144.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout:Installed: 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:29:49.511 INFO:teuthology.orchestra.run.smithi144.stdout:Complete! 2024-09-20T23:29:49.866 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T23:29:49.948 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:49.949 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T23:29:49.949 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-20T23:29:49.949 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-20T23:29:49.949 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-20T23:29:49.950 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-09-20T23:29:49.950 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi144.front.sepia.ceph.com, path=None, version=distro) 2024-09-20T23:29:49.950 DEBUG:teuthology.orchestra.run.smithi144:> rpm -q kernel | sort -rV | head -n 1 2024-09-20T23:29:50.008 INFO:teuthology.orchestra.run.smithi144.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-20T23:29:50.008 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-20T23:29:50.008 DEBUG:teuthology.orchestra.run.smithi144:> sudo rpm -qi grub2-tools 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Name : grub2-tools 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Epoch : 1 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Version : 2.06 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Release : 82.el9 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Architecture: x86_64 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-20T23:29:50.043 INFO:teuthology.orchestra.run.smithi144.stdout:Group : Unspecified 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Size : 8274670 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:License : GPLv3+ 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Packager : builder@centos.org 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Vendor : CentOS 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Summary : Support tools for GRUB. 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:Description : 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-20T23:29:50.044 INFO:teuthology.orchestra.run.smithi144.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-20T23:29:50.045 INFO:teuthology.orchestra.run.smithi144.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-20T23:29:50.045 INFO:teuthology.orchestra.run.smithi144.stdout:hardware devices. 2024-09-20T23:29:50.045 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:29:50.045 INFO:teuthology.orchestra.run.smithi144.stdout:This subpackage provides tools for support of all platforms. 2024-09-20T23:29:50.045 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-20T23:29:50.045 INFO:teuthology.task.kernel:Updating grub on smithi144 to boot 5.14.0-508.el9.x86_64 2024-09-20T23:29:50.046 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-20T23:29:50.234 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T23:29:50.252 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-20T23:29:50.253 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-20T23:29:50.268 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.8ssmpCo1Wv 2024-09-20T23:29:50.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.8ssmpCo1Wv 2024-09-20T23:29:50.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.8ssmpCo1Wv 2024-09-20T23:29:50.484 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.8ssmpCo1Wv is 6KB 2024-09-20T23:29:50.534 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.8ssmpCo1Wv 2024-09-20T23:29:50.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T23:29:50.620 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T23:29:50.620 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T23:29:50.620 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T23:29:50.620 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T23:29:50.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T23:29:50.740 INFO:teuthology.orchestra.run.smithi144.stderr:Generating grub configuration file ... 2024-09-20T23:29:50.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-20T23:29:52.022 INFO:teuthology.orchestra.run.smithi144.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-20T23:29:52.051 INFO:teuthology.orchestra.run.smithi144.stderr:done 2024-09-20T23:29:52.056 DEBUG:teuthology.orchestra.run.smithi144:> mktemp 2024-09-20T23:29:52.072 INFO:teuthology.orchestra.run.smithi144.stdout:/tmp/tmp.ioRRluLRRI 2024-09-20T23:29:52.073 DEBUG:teuthology.orchestra.run.smithi144:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ioRRluLRRI 2024-09-20T23:29:52.140 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 0666 /tmp/tmp.ioRRluLRRI 2024-09-20T23:29:52.278 DEBUG:teuthology.orchestra.remote:smithi144:/tmp/tmp.ioRRluLRRI is 6KB 2024-09-20T23:29:52.327 DEBUG:teuthology.orchestra.run.smithi144:> rm -fr /tmp/tmp.ioRRluLRRI 2024-09-20T23:29:52.342 DEBUG:teuthology.orchestra.run.smithi144:> sudo /bin/ls /boot/loader/entries || true 2024-09-20T23:29:52.410 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-20T23:29:52.410 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-20T23:29:52.410 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-20T23:29:52.410 INFO:teuthology.orchestra.run.smithi144.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-20T23:29:52.411 DEBUG:teuthology.orchestra.run.smithi144:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-20T23:29:52.571 DEBUG:teuthology.orchestra.run.smithi144:> sudo shutdown -r now 2024-09-20T23:30:20.787 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-09-20T23:30:20.787 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:30:20.788 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:30:22.574 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-09-20T23:30:22.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:30:22.575 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:30:39.215 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-20T23:30:41.007 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.144 2024-09-20T23:30:48.223 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:30:48.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:30:50.009 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:30:50.010 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:30:51.375 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.137 2024-09-20T23:31:03.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:31:03.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:31:35.070 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-20T23:31:35.782 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:31:35.782 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-508.el9.x86_64"... 2024-09-20T23:31:35.783 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-20T23:31:35.797 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-508.el9.x86_64 2024-09-20T23:31:35.797 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T23:31:35.797 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T23:31:35.797 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T23:31:36.798 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-09-20T23:31:36.798 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T23:31:36.865 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-20T23:31:36.889 DEBUG:teuthology.parallel:result is None 2024-09-20T23:31:50.012 DEBUG:teuthology.orchestra.remote:timed out 2024-09-20T23:32:02.013 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:32:02.014 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:32:02.314 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-20T23:32:02.700 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:32:02.700 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-508.el9.x86_64"... 2024-09-20T23:32:02.700 DEBUG:teuthology.orchestra.run.smithi144:> uname -r 2024-09-20T23:32:02.716 INFO:teuthology.orchestra.run.smithi144.stdout:5.14.0-508.el9.x86_64 2024-09-20T23:32:02.716 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-20T23:32:02.716 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-20T23:32:02.716 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-20T23:32:03.717 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-09-20T23:32:03.717 DEBUG:teuthology.orchestra.run.smithi144:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-20T23:32:03.783 INFO:teuthology.orchestra.run.smithi144.stdout:ttyS1 2024-09-20T23:32:03.816 DEBUG:teuthology.parallel:result is None 2024-09-20T23:32:03.816 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-20T23:32:03.829 INFO:teuthology.task.internal:Creating test directory... 2024-09-20T23:32:03.830 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T23:32:03.832 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-20T23:32:03.850 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-20T23:32:03.906 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-20T23:32:04.027 INFO:teuthology.task.internal:Creating archive directory... 2024-09-20T23:32:04.028 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T23:32:04.030 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-20T23:32:04.062 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-20T23:32:04.255 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-20T23:32:04.256 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T23:32:04.258 DEBUG:teuthology.orchestra.run.smithi144:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-20T23:32:04.291 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T23:32:04.298 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T23:32:04.303 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T23:32:04.309 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-20T23:32:04.311 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-20T23:32:04.438 INFO:teuthology.task.internal:Configuring sudo... 2024-09-20T23:32:04.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T23:32:04.441 DEBUG:teuthology.orchestra.run.smithi144:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-20T23:32:04.469 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-20T23:32:04.589 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-20T23:32:04.589 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T23:32:04.592 DEBUG:teuthology.orchestra.run.smithi144:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-20T23:32:04.609 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T23:32:04.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T23:32:04.741 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T23:32:04.822 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T23:32:04.867 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:32:04.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T23:32:04.934 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T23:32:04.974 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-20T23:32:05.035 DEBUG:teuthology.orchestra.run.smithi144:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T23:32:05.117 DEBUG:teuthology.orchestra.run.smithi144:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-20T23:32:05.163 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:32:05.163 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-20T23:32:05.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-20T23:32:05.233 DEBUG:teuthology.orchestra.run.smithi144:> sudo service rsyslog restart 2024-09-20T23:32:05.275 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T23:32:05.313 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-20T23:32:05.732 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-20T23:32:05.740 INFO:teuthology.task.internal:Starting timer... 2024-09-20T23:32:05.740 INFO:teuthology.run_tasks:Running task pcp... 2024-09-20T23:32:05.800 INFO:teuthology.run_tasks:Running task selinux... 2024-09-20T23:32:05.909 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-09-20T23:32:05.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-09-20T23:32:05.967 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-09-20T23:32:05.969 DEBUG:teuthology.orchestra.run.smithi144:> sudo service auditd rotate 2024-09-20T23:32:06.014 INFO:teuthology.orchestra.run.smithi144.stdout:Rotating logs: 2024-09-20T23:32:06.016 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-20T23:32:06.016 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-09-20T23:32:06.041 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-09-20T23:32:06.041 DEBUG:teuthology.orchestra.run.smithi144:> /usr/sbin/getenforce 2024-09-20T23:32:06.078 INFO:teuthology.orchestra.run.smithi144.stdout:Permissive 2024-09-20T23:32:06.078 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi137.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi144.front.sepia.ceph.com': 'permissive'} 2024-09-20T23:32:06.078 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T23:32:06.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:32:06.110 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-20T23:32:06.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-20T23:32:06.147 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-20T23:32:06.147 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-09-20T23:32:06.188 DEBUG:teuthology.orchestra.run.smithi144:> sudo /usr/sbin/setenforce permissive 2024-09-20T23:32:06.227 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-20T23:32:06.237 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-20T23:32:06.247 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-20T23:32:06.248 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com,smithi144.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-20T23:35:48.946 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com'), Remote(name='ubuntu@smithi144.front.sepia.ceph.com')] 2024-09-20T23:35:48.947 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:35:48.948 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:35:49.021 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-20T23:35:49.093 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-20T23:35:49.093 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:35:49.094 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi144.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-20T23:35:49.166 DEBUG:teuthology.orchestra.run.smithi144:> true 2024-09-20T23:35:49.238 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi144.front.sepia.ceph.com' 2024-09-20T23:35:49.238 INFO:teuthology.run_tasks:Running task clock... 2024-09-20T23:35:49.248 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-20T23:35:49.248 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T23:35:49.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T23:35:49.251 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-20T23:35:49.251 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-20T23:35:49.277 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T23:35:49.290 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T23:35:49.307 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-20T23:35:49.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-09-20T23:35:49.321 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-20T23:35:49.325 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-09-20T23:35:49.339 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T23:35:49.346 INFO:teuthology.orchestra.run.smithi144.stderr:sudo: ntpd: command not found 2024-09-20T23:35:49.353 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T23:35:49.357 INFO:teuthology.orchestra.run.smithi144.stdout:506 Cannot talk to daemon 2024-09-20T23:35:49.371 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-20T23:35:49.385 INFO:teuthology.orchestra.run.smithi144.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-20T23:35:49.410 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-20T23:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T23:35:49.413 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-20T23:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.414 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.447 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-09-20T23:35:49.451 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-20T23:35:49.451 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-09-20T23:35:49.451 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.452 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.452 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.452 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-20T23:35:49.452 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-20T23:35:49.545 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_REFRESH_FAILED'], 'log-only-match': ['CEPHADM_'], 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'} 2024-09-20T23:35:49.545 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-09-20T23:35:49.545 INFO:tasks.cephadm:Cluster fsid is 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:35:49.546 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-20T23:35:49.546 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-09-20T23:35:49.546 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi137': '172.21.15.137', 'mon.smithi144': '172.21.15.144'} 2024-09-20T23:35:49.546 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-20T23:35:49.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-09-20T23:35:49.572 DEBUG:teuthology.orchestra.run.smithi144:> sudo hostname $(hostname -s) 2024-09-20T23:35:49.599 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-09-20T23:35:49.599 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.731 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 20 23:35 /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.731 DEBUG:teuthology.orchestra.run.smithi144:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.852 INFO:teuthology.orchestra.run.smithi144.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Sep 20 23:35 /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.852 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.868 DEBUG:teuthology.orchestra.run.smithi144:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-20T23:35:49.891 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-09-20T23:35:49.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-20T23:35:49.911 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-09-20T23:35:51.512 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T23:35:51.571 INFO:teuthology.orchestra.run.smithi144.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-20T23:36:03.659 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:36:03.873 INFO:teuthology.orchestra.run.smithi144.stdout:{ 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: "repo_digests": [ 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout: ] 2024-09-20T23:36:03.874 INFO:teuthology.orchestra.run.smithi144.stdout:} 2024-09-20T23:36:03.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-09-20T23:36:03.921 DEBUG:teuthology.orchestra.run.smithi144:> sudo mkdir -p /etc/ceph 2024-09-20T23:36:03.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-09-20T23:36:03.986 DEBUG:teuthology.orchestra.run.smithi144:> sudo chmod 777 /etc/ceph 2024-09-20T23:36:04.015 INFO:tasks.cephadm:Writing seed config... 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-20T23:36:04.016 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-20T23:36:04.017 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:36:04.017 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-20T23:36:04.041 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 1150c2f6-77a9-11ef-baf5-efdc52797490 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-20T23:36:04.041 DEBUG:teuthology.orchestra.run.smithi137:mon.smithi137> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service 2024-09-20T23:36:04.083 INFO:tasks.cephadm:Bootstrapping... 2024-09-20T23:36:04.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.137 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:36:04.273 INFO:teuthology.orchestra.run.smithi137.stderr:-------------------------------------------------------------------------------- 2024-09-20T23:36:04.273 INFO:teuthology.orchestra.run.smithi137.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '1150c2f6-77a9-11ef-baf5-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.137', '--skip-admin-label'] 2024-09-20T23:36:04.294 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: 5.2.2 2024-09-20T23:36:04.297 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying podman|docker is present... 2024-09-20T23:36:04.317 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: 5.2.2 2024-09-20T23:36:04.319 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying lvm2 is present... 2024-09-20T23:36:04.319 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying time synchronization is in place... 2024-09-20T23:36:04.326 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T23:36:04.333 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:36:04.340 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: enabled 2024-09-20T23:36:04.346 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: active 2024-09-20T23:36:04.346 INFO:teuthology.orchestra.run.smithi137.stderr:Unit chronyd.service is enabled and running 2024-09-20T23:36:04.346 INFO:teuthology.orchestra.run.smithi137.stderr:Repeating the final host check... 2024-09-20T23:36:04.366 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: 5.2.2 2024-09-20T23:36:04.369 INFO:teuthology.orchestra.run.smithi137.stderr:podman (/bin/podman) version 5.2.2 is present 2024-09-20T23:36:04.369 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl is present 2024-09-20T23:36:04.369 INFO:teuthology.orchestra.run.smithi137.stderr:lvcreate is present 2024-09-20T23:36:04.375 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-09-20T23:36:04.381 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:36:04.388 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: enabled 2024-09-20T23:36:04.394 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: active 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Unit chronyd.service is enabled and running 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Host looks OK 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Cluster fsid: 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Acquiring lock 140078595917712 on /run/cephadm/1150c2f6-77a9-11ef-baf5-efdc52797490.lock 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Lock 140078595917712 acquired on /run/cephadm/1150c2f6-77a9-11ef-baf5-efdc52797490.lock 2024-09-20T23:36:04.395 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying IP 172.21.15.137 port 3300 ... 2024-09-20T23:36:04.396 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying IP 172.21.15.137 port 6789 ... 2024-09-20T23:36:04.396 INFO:teuthology.orchestra.run.smithi137.stderr:Base mon IP is 172.21.15.137, final addrv is [v2:172.21.15.137:3300,v1:172.21.15.137:6789] 2024-09-20T23:36:04.398 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.137 metric 100 2024-09-20T23:36:04.398 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.137 metric 100 2024-09-20T23:36:04.401 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-09-20T23:36:04.401 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-20T23:36:04.401 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-20T23:36:04.403 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-20T23:36:04.404 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-09-20T23:36:04.404 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-20T23:36:04.404 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-20T23:36:04.404 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe88:8137/64 scope link noprefixroute 2024-09-20T23:36:04.404 INFO:teuthology.orchestra.run.smithi137.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-09-20T23:36:04.405 INFO:teuthology.orchestra.run.smithi137.stderr:Mon IP `172.21.15.137` is in CIDR network `172.21.0.0/20` 2024-09-20T23:36:04.405 INFO:teuthology.orchestra.run.smithi137.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-20T23:36:04.405 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-09-20T23:36:04.504 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Getting image source signatures 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-09-20T23:36:04.843 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-09-20T23:36:04.844 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-20T23:36:04.996 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: Writing manifest to image destination 2024-09-20T23:36:05.115 INFO:teuthology.orchestra.run.smithi137.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-09-20T23:36:05.500 INFO:teuthology.orchestra.run.smithi137.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-20T23:36:06.040 INFO:teuthology.orchestra.run.smithi137.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-09-20T23:36:06.040 INFO:teuthology.orchestra.run.smithi137.stderr:Extracting ceph user uid/gid from container image... 2024-09-20T23:36:06.405 INFO:teuthology.orchestra.run.smithi137.stderr:stat: 167 167 2024-09-20T23:36:06.870 INFO:teuthology.orchestra.run.smithi137.stderr:Creating initial keys... 2024-09-20T23:36:07.249 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph-authtool: AQDnBu5mAQfvCRAAs9l4lgg6eS3qw6w/lhcZww== 2024-09-20T23:36:08.150 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph-authtool: AQDoBu5mHbBYAxAAUFWJRHHdCOF4q2hX3jehXg== 2024-09-20T23:36:09.172 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph-authtool: AQDpBu5mjTgjBRAAWaMdYLX8Kv7Iw8BB2rM3NA== 2024-09-20T23:36:09.771 INFO:teuthology.orchestra.run.smithi137.stderr:Creating initial monmap... 2024-09-20T23:36:10.103 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T23:36:10.103 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-09-20T23:36:10.103 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:10.103 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:monmaptool for smithi137 [v2:172.21.15.137:3300,v1:172.21.15.137:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:setting min_mon_release = octopus 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: set fsid to 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:36:10.517 INFO:teuthology.orchestra.run.smithi137.stderr:Creating mon... 2024-09-20T23:36:11.392 INFO:teuthology.orchestra.run.smithi137.stderr:create mon.smithi137 on 2024-09-20T23:36:11.596 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-20T23:36:11.742 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target. 2024-09-20T23:36:11.742 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target → /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target. 2024-09-20T23:36:12.043 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Failed to reset failed state of unit ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service: Unit ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service not loaded. 2024-09-20T23:36:12.050 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Created symlink /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target.wants/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service → /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@.service. 2024-09-20T23:36:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:12 smithi137 podman[23339]: 2024-09-20 23:36:12.337197872 +0000 UTC m=+0.016410747 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-20T23:36:13.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:13 smithi137 podman[23339]: 2024-09-20 23:36:13.176649421 +0000 UTC m=+0.855862297 container create c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , version=8, io.buildah.version=1.19.8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc.) 2024-09-20T23:36:15.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:14 smithi137 podman[23339]: 2024-09-20 23:36:14.99786633 +0000 UTC m=+2.677079219 container init c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, com.redhat.component=centos-stream-container, name=centos-stream, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, architecture=x86_64, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, release=754, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-09-20T23:36:15.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:15 smithi137 podman[23339]: 2024-09-20 23:36:15.001996451 +0000 UTC m=+2.681209325 container start c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, name=centos-stream, version=8, distribution-scope=public, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, vcs-type=git, io.openshift.tags=base centos centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, vendor=Red Hat, Inc., GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD) 2024-09-20T23:36:15.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:15 smithi137 ceph-mon[23357]: mkfs 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:15.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:15 smithi137 ceph-mon[23357]: mon.smithi137 is new leader, mons smithi137 in quorum (ranks 0) 2024-09-20T23:36:15.835 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: disabled 2024-09-20T23:36:15.843 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:36:15.843 INFO:teuthology.orchestra.run.smithi137.stderr:firewalld.service is not enabled 2024-09-20T23:36:15.843 INFO:teuthology.orchestra.run.smithi137.stderr:Not possible to enable service . firewalld.service is not available 2024-09-20T23:36:15.844 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mon to start... 2024-09-20T23:36:15.844 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mon... 2024-09-20T23:36:16.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:15 smithi137 bash[23339]: c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:15 smithi137 systemd[1]: Started Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:16 smithi137 ceph-mon[23357]: mon.smithi137 is new leader, mons smithi137 in quorum (ranks 0) 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:16 smithi137 ceph-mon[23357]: monmap e1: 1 mons at {smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:16 smithi137 ceph-mon[23357]: fsmap 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:16 smithi137 ceph-mon[23357]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T23:36:16.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:16 smithi137 ceph-mon[23357]: mgrmap e1: no daemons active 2024-09-20T23:36:19.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:19 smithi137 ceph-mon[23357]: from='client.? 172.21.15.137:0/2662718276' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-20T23:36:19.649 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: cluster: 2024-09-20T23:36:19.649 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: id: 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:19.649 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: services: 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi137 (age 3s) 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mgr: no daemons active 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: data: 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: pgs: 2024-09-20T23:36:19.650 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:23.865 INFO:teuthology.orchestra.run.smithi137.stderr:mon is available 2024-09-20T23:36:23.865 INFO:teuthology.orchestra.run.smithi137.stderr:Assimilating anything we can from ceph.conf... 2024-09-20T23:36:24.369 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:24.369 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [global] 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: fsid = 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.137:3300,v1:172.21.15.137:6789] 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-20T23:36:24.370 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [mgr] 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [osd] 2024-09-20T23:36:24.371 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-20T23:36:24.372 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-20T23:36:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:24 smithi137 ceph-mon[23357]: from='client.? 172.21.15.137:0/1807417548' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T23:36:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:24 smithi137 ceph-mon[23357]: from='client.? 172.21.15.137:0/1807417548' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-20T23:36:24.929 INFO:teuthology.orchestra.run.smithi137.stderr:Generating new minimal ceph.conf... 2024-09-20T23:36:26.002 INFO:teuthology.orchestra.run.smithi137.stderr:Restarting the monitor... 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 systemd[1]: Stopping Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 ceph-mon[23357]: from='client.? 172.21.15.137:0/291553780' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 bash[23576]: Error: no container with name or ID "ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon.smithi137" found: no such container 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[23353]: 2024-09-20T23:36:26.184+0000 7f7773627700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi137 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[23353]: 2024-09-20T23:36:26.184+0000 7f7773627700 -1 mon.smithi137@0(leader) e1 *** Got Signal Terminated *** 2024-09-20T23:36:26.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 podman[23588]: 2024-09-20 23:36:26.419788472 +0000 UTC m=+0.360132961 container died c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, vendor=Red Hat, Inc., version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-09-20T23:36:26.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 podman[23588]: 2024-09-20 23:36:26.759294549 +0000 UTC m=+0.699639025 container cleanup c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, vendor=Red Hat, Inc., distribution-scope=public, version=8, io.openshift.expose-services=, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, release=754, GIT_CLEAN=True, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0) 2024-09-20T23:36:26.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 bash[23588]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 podman[23601]: 2024-09-20 23:36:26.96023129 +0000 UTC m=+0.531064085 container remove c8630d4a2fec90552fa70c864ae11cd9032faed0d64df1d7b671ceb6a9a962cb (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, io.openshift.tags=base centos centos-stream, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, GIT_BRANCH=HEAD, ceph=True, version=8, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754) 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 bash[23616]: Error: no container with name or ID "ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon.smithi137" found: no such container 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service: Deactivated successfully. 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 systemd[1]: Stopped Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:26 smithi137 systemd[1]: Starting Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:36:27.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 podman[23679]: 2024-09-20 23:36:27.10892211 +0000 UTC m=+0.016474758 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-09-20T23:36:27.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 podman[23679]: 2024-09-20 23:36:27.219295673 +0000 UTC m=+0.126848310 container create fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, com.redhat.component=centos-stream-container, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , GIT_CLEAN=True, name=centos-stream, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8, io.openshift.tags=base centos centos-stream, architecture=x86_64, vcs-type=git, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 podman[23679]: 2024-09-20 23:36:27.421642571 +0000 UTC m=+0.329195215 container init fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, release=754, RELEASE=HEAD, name=centos-stream, version=8, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 podman[23679]: 2024-09-20 23:36:27.42473001 +0000 UTC m=+0.332282648 container start fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, architecture=x86_64, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.buildah.version=1.19.8, ceph=True, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: pidfile_write: ignore empty --pid-file 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: load: jerasure load: lrc 2024-09-20T23:36:27.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: RocksDB version: 6.15.5 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Compile date Apr 18 2022 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: DB SUMMARY 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: DB Session ID: FGMRNWL5B9PC5876QREQ 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: CURRENT file: CURRENT 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi137/store.db dir, Total Num: 1, files: 000008.sst 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi137/store.db: 000010.log size: 84036 ; 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.error_if_exists: 0 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.create_if_missing: 0 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.paranoid_checks: 1 2024-09-20T23:36:27.499 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.env: 0x55b5204c6860 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.fs: Posix File System 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.info_log: 0x55b5216affa0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.statistics: (nil) 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.use_fsync: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_log_file_size: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_fallocate: 1 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T23:36:27.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.use_direct_reads: 0 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.db_log_dir: 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi137/store.db 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T23:36:27.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.write_buffer_manager: 0x55b5217a02d0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T23:36:27.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.unordered_write: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.row_cache: None 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.wal_filter: None 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.preserve_deletes: 0 2024-09-20T23:36:27.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.two_write_queues: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.atomic_flush: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.log_readahead_size: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T23:36:27.504 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_background_jobs: 2 2024-09-20T23:36:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_background_compactions: -1 2024-09-20T23:36:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_subcompactions: 1 2024-09-20T23:36:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T23:36:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T23:36:27.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T23:36:27.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_open_files: -1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_background_flushes: -1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Compression algorithms supported: 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kZSTD supported: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kXpressCompression supported: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kLZ4Compression supported: 1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kBZip2Compression supported: 0 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kZlibCompression supported: 1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: kSnappyCompression supported: 1 2024-09-20T23:36:27.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000009 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.merge_operator: 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_filter: None 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b52167bd00) 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T23:36:27.508 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_type: 0 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_index_type: 0 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_shortening: 1 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: hash_index_allow_collision: 1 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: checksum: 1 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: no_block_cache: 0 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache: 0x55b5216e6170 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_options: 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: capacity : 536870912 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_shard_bits : 4 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: strict_capacity_limit : 0 2024-09-20T23:36:27.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_compressed: (nil) 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: persistent_cache: (nil) 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size: 4096 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size_deviation: 10 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_restart_interval: 16 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_block_restart_interval: 1 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_block_size: 4096 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: partition_filters: 0 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: use_delta_encoding: 1 2024-09-20T23:36:27.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: whole_key_filtering: 1 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: verify_compression: 0 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: format_version: 4 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_index_compression: 1 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_align: 0 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression: NoCompression 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.num_levels: 7 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T23:36:27.511 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T23:36:27.512 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T23:36:27.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.arena_block_size: 4194304 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T23:36:27.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.table_properties_collectors: 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.inplace_update_support: 0 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.bloom_locality: 0 2024-09-20T23:36:27.515 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.max_successive_merges: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.ttl: 2592000 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.enable_blob_files: false 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.min_blob_size: 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-09-20T23:36:27.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726875387451236, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726875387451982, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 80121, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 78366, "index_size": 243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9946, "raw_average_key_size": 46, "raw_value_size": 72603, "raw_average_value_size": 340, "num_data_blocks": 11, "num_entries": 213, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1726875387, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "959a3a4d-5c4c-4cc0-a40c-4ff068cf2f4d", "db_session_id": "FGMRNWL5B9PC5876QREQ"}} 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726875387452361, "job": 1, "event": "recovery_finished"} 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi137/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55b5216cca80 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: DB pointer 0x55b5216dc000 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** DB Stats ** 2024-09-20T23:36:27.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 2/0 80.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 2/0 80.04 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.518 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 25.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 25.06 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T23:36:27.519 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 2/0 80.04 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 2/0 80.04 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 150.7 0.00 0.00 1 0.001 0 0 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:36:27.520 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 24.81 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: starting mon.smithi137 rank 0 at public addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] at bind addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi137 fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???) e1 preinit fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:27.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).mds e1 new map 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).mds e1 print_map 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: e1 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: legacy client fscid: -1 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: No filesystems configured 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expand map: {default=false} 2024-09-20T23:36:27.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta from 'false' to 'false' 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expanded map: {default=false} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expand map: {default=info} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta from 'info' to 'info' 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expanded map: {default=info} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expand map: {default=daemon} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta from 'daemon' to 'daemon' 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expanded map: {default=daemon} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expand map: {default=debug} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta from 'debug' to 'debug' 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: expand_channel_meta expanded map: {default=debug} 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 bash[23679]: fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 2024-09-20T23:36:27.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 systemd[1]: Started Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:36:27.523 INFO:teuthology.orchestra.run.smithi137.stderr:Setting mon public_network to 172.21.0.0/20 2024-09-20T23:36:27.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mon.smithi137 is new leader, mons smithi137 in quorum (ranks 0) 2024-09-20T23:36:27.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: monmap e1: 1 mons at {smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0]} 2024-09-20T23:36:27.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: fsmap 2024-09-20T23:36:27.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: osdmap e1: 0 total, 0 up, 0 in 2024-09-20T23:36:27.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:27 smithi137 ceph-mon[23698]: mgrmap e1: no daemons active 2024-09-20T23:36:28.532 INFO:teuthology.orchestra.run.smithi137.stderr:Wrote config to /etc/ceph/ceph.conf 2024-09-20T23:36:28.532 INFO:teuthology.orchestra.run.smithi137.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:36:28.532 INFO:teuthology.orchestra.run.smithi137.stderr:Creating mgr... 2024-09-20T23:36:28.532 INFO:teuthology.orchestra.run.smithi137.stderr:Verifying port 9283 ... 2024-09-20T23:36:28.703 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Failed to reset failed state of unit ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mgr.smithi137.eurecx.service: Unit ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mgr.smithi137.eurecx.service not loaded. 2024-09-20T23:36:28.711 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: Created symlink /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490.target.wants/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mgr.smithi137.eurecx.service → /etc/systemd/system/ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@.service. 2024-09-20T23:36:29.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:28 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/478351040' entity='client.admin' 2024-09-20T23:36:30.028 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: disabled 2024-09-20T23:36:30.037 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:36:30.038 INFO:teuthology.orchestra.run.smithi137.stderr:firewalld.service is not enabled 2024-09-20T23:36:30.038 INFO:teuthology.orchestra.run.smithi137.stderr:Not possible to enable service . firewalld.service is not available 2024-09-20T23:36:30.048 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: disabled 2024-09-20T23:36:30.056 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:36:30.056 INFO:teuthology.orchestra.run.smithi137.stderr:firewalld.service is not enabled 2024-09-20T23:36:30.057 INFO:teuthology.orchestra.run.smithi137.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-09-20T23:36:30.057 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mgr to start... 2024-09-20T23:36:30.057 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mgr... 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsid": "1150c2f6-77a9-11ef-baf5-efdc52797490", 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "health": { 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T23:36:30.629 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 0 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "smithi137" 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_age": 3, 2024-09-20T23:36:30.630 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T23:36:30.631 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T23:36:30.632 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T23:36:30.635 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "available": false, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "iostat", 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "nfs", 2024-09-20T23:36:30.636 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "restful" 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modified": "2024-09-20T23:36:15.045438+0000", 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T23:36:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:36:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:30 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3208024620' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T23:36:31.171 INFO:teuthology.orchestra.run.smithi137.stderr:mgr not available, waiting (1/15)... 2024-09-20T23:36:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:33.686 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsid": "1150c2f6-77a9-11ef-baf5-efdc52797490", 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "health": { 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 0 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "smithi137" 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_age": 6, 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T23:36:33.687 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T23:36:33.688 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "available": false, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T23:36:33.689 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "iostat", 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "nfs", 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "restful" 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modified": "2024-09-20T23:36:15.045438+0000", 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T23:36:33.690 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:36:33.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:33 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1065756934' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T23:36:34.335 INFO:teuthology.orchestra.run.smithi137.stderr:mgr not available, waiting (2/15)... 2024-09-20T23:36:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: Activating manager daemon smithi137.eurecx 2024-09-20T23:36:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: mgrmap e2: smithi137.eurecx(active, starting, since 0.00277941s) 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:36.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:36 smithi137 ceph-mon[23698]: from='mgr.14100 172.21.15.137:0/3500603503' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsid": "1150c2f6-77a9-11ef-baf5-efdc52797490", 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "health": { 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "checks": {}, 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mutes": [] 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum": [ 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 0 2024-09-20T23:36:37.612 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_names": [ 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "smithi137" 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "quorum_age": 10, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "monmap": { 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_mons": 1 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osdmap": { 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_osds": 0, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-09-20T23:36:37.613 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-09-20T23:36:37.614 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-09-20T23:36:37.616 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-09-20T23:36:37.616 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgmap": { 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_pools": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_objects": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "bytes_total": 0 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "fsmap": { 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "by_rank": [], 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "up:standby": 0 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.617 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mgrmap": { 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "available": true, 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modules": [ 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "iostat", 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "nfs", 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "restful" 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ], 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "servicemap": { 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 1, 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "modified": "2024-09-20T23:36:15.045438+0000", 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "services": {} 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: }, 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "progress_events": {} 2024-09-20T23:36:37.618 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:36:38.200 INFO:teuthology.orchestra.run.smithi137.stderr:mgr is available 2024-09-20T23:36:38.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:38 smithi137 ceph-mon[23698]: mgrmap e3: smithi137.eurecx(active, since 1.00698s) 2024-09-20T23:36:38.293 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:38 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3237132745' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-20T23:36:38.809 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [global] 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: fsid = 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [mgr] 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: [osd] 2024-09-20T23:36:38.810 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-09-20T23:36:38.811 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-09-20T23:36:39.415 INFO:teuthology.orchestra.run.smithi137.stderr:Enabling cephadm module... 2024-09-20T23:36:39.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:39 smithi137 ceph-mon[23698]: mgrmap e4: smithi137.eurecx(active, since 2s) 2024-09-20T23:36:39.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:39 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1567514233' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-20T23:36:40.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:40 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1277523042' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-20T23:36:41.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:41 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1277523042' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-20T23:36:41.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:41 smithi137 ceph-mon[23698]: mgrmap e5: smithi137.eurecx(active, since 4s) 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 5, 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "available": true, 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "active_name": "smithi137.eurecx", 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-20T23:36:41.742 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:36:42.303 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for the mgr to restart... 2024-09-20T23:36:42.303 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mgr epoch 5... 2024-09-20T23:36:42.603 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:42 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1435477510' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T23:36:46.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:46 smithi137 ceph-mon[23698]: Active manager daemon smithi137.eurecx restarted 2024-09-20T23:36:46.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:46 smithi137 ceph-mon[23698]: Activating manager daemon smithi137.eurecx 2024-09-20T23:36:46.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:46 smithi137 ceph-mon[23698]: osdmap e2: 0 total, 0 up, 0 in 2024-09-20T23:36:47.605 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:36:47.605 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-09-20T23:36:47.605 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "initialized": true 2024-09-20T23:36:47.605 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:36:47.922 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: mgrmap e6: smithi137.eurecx(active, starting, since 0.0527269s) 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:36:47.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:47 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:36:48.166 INFO:teuthology.orchestra.run.smithi137.stderr:mgr epoch 5 is available 2024-09-20T23:36:48.167 INFO:teuthology.orchestra.run.smithi137.stderr:Setting orchestrator backend to cephadm... 2024-09-20T23:36:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: mgrmap e7: smithi137.eurecx(active, since 1.05768s) 2024-09-20T23:36:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: [20/Sep/2024:23:36:48] ENGINE Bus STARTING 2024-09-20T23:36:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: [20/Sep/2024:23:36:48] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:36:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: [20/Sep/2024:23:36:48] ENGINE Bus STARTED 2024-09-20T23:36:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:48 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:49.917 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: value unchanged 2024-09-20T23:36:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:49 smithi137 ceph-mon[23698]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:49 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:49 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:49 smithi137 ceph-mon[23698]: mgrmap e8: smithi137.eurecx(active, since 2s) 2024-09-20T23:36:50.396 INFO:teuthology.orchestra.run.smithi137.stderr:Generating ssh key... 2024-09-20T23:36:51.108 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:50 smithi137 ceph-mon[23698]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:52 smithi137 ceph-mon[23698]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:52.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:52 smithi137 ceph-mon[23698]: Generating ssh key... 2024-09-20T23:36:52.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:52 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:52.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:52 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:52.454 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcyKknEYST9WHyVl2PDD88GRYVUB+/LZ8g4mDEdvjjmnnqsbkJ1KL2uHEqJpXy/ODpI8U9VoLz5zD4FK1xkJP8yKMYyhy5FmpjlR+0y4bjg3zOhLlHH7GXW6cCJLMECnQOMVrkBR0Khxrv80LpGw8QhSC05mQY+DVvf7HkPph8M9NHGbz4oKBzLedLSeAcB+UEhAt2gWfOMYe2JEV1CH8pGIp9Y4RWJii6b3HEBu7rc0YgbSkTXYvGxvbvzqfeRHBxoROyh2qQiPw5Gv13r3avpSG3IwDxPzAFdCfDX3tJE67wV/Pgq1oizF7day36DZJ6HsSCFH1rc5ikOSq6FBow3P5vRESH3WkM1w46zKcvrJUb9Mfy7niA4BwZLwzlu9CUK9/cPBHXe8v2CAEgdK1M1cgpOWK+RoTXXTGneIfm4kn3mMAs8ZqxUNtfP1xcdqtGheKs+9FBA29n+7PI+dbbRkbntjmuD7iFZbUcUPtulTK14FVQr0Rr+5qNjQFW+k= ceph-1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:36:52.983 INFO:teuthology.orchestra.run.smithi137.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-20T23:36:52.983 INFO:teuthology.orchestra.run.smithi137.stderr:Adding key to root@localhost authorized_keys... 2024-09-20T23:36:52.986 INFO:teuthology.orchestra.run.smithi137.stderr:Adding host smithi137... 2024-09-20T23:36:53.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:53 smithi137 ceph-mon[23698]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:54.423 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Added host 'smithi137' with addr '172.21.15.137' 2024-09-20T23:36:54.982 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying mon service with default placement... 2024-09-20T23:36:55.594 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:55 smithi137 ceph-mon[23698]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "addr": "172.21.15.137", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:55.594 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:55 smithi137 ceph-mon[23698]: Deploying cephadm binary to smithi137 2024-09-20T23:36:55.594 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:55 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:55.594 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:55 smithi137 ceph-mon[23698]: Added host smithi137 2024-09-20T23:36:55.594 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:55 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:36:55.595 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled mon update... 2024-09-20T23:36:56.198 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying mgr service with default placement... 2024-09-20T23:36:56.854 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:56 smithi137 ceph-mon[23698]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:56.855 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:56 smithi137 ceph-mon[23698]: Saving service mon spec with placement count:5 2024-09-20T23:36:56.855 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:56 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:56.855 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-09-20T23:36:57.607 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying crash service with default placement... 2024-09-20T23:36:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:57 smithi137 ceph-mon[23698]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:57 smithi137 ceph-mon[23698]: Saving service mgr spec with placement count:2 2024-09-20T23:36:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:57 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:36:58.238 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled crash update... 2024-09-20T23:36:58.814 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying prometheus service with default placement... 2024-09-20T23:36:59.389 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-09-20T23:36:59.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:59 smithi137 ceph-mon[23698]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:36:59.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:59 smithi137 ceph-mon[23698]: Saving service crash spec with placement * 2024-09-20T23:36:59.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:36:59 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:00.098 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying grafana service with default placement... 2024-09-20T23:37:00.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:00 smithi137 ceph-mon[23698]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:00.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:00 smithi137 ceph-mon[23698]: Saving service prometheus spec with placement count:1 2024-09-20T23:37:00.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:00 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:00.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:00 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:00.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:00 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:00.792 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-09-20T23:37:01.899 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying node-exporter service with default placement... 2024-09-20T23:37:02.061 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:01 smithi137 ceph-mon[23698]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:02.062 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:01 smithi137 ceph-mon[23698]: Saving service grafana spec with placement count:1 2024-09-20T23:37:02.062 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:01 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:02.062 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:01 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:02.539 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-09-20T23:37:03.098 INFO:teuthology.orchestra.run.smithi137.stderr:Deploying alertmanager service with default placement... 2024-09-20T23:37:03.566 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:03 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:03.566 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:03 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:03.651 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-09-20T23:37:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:04 smithi137 ceph-mon[23698]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:04 smithi137 ceph-mon[23698]: Saving service node-exporter spec with placement * 2024-09-20T23:37:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:04 smithi137 ceph-mon[23698]: from='mgr.14118 172.21.15.137:0/1939697033' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:05.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:05 smithi137 ceph-mon[23698]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:05.592 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:05 smithi137 ceph-mon[23698]: Saving service alertmanager spec with placement count:1 2024-09-20T23:37:06.877 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:06 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3374488333' entity='client.admin' 2024-09-20T23:37:07.457 INFO:teuthology.orchestra.run.smithi137.stderr:Enabling the dashboard module... 2024-09-20T23:37:08.165 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:07 smithi137 ceph-mon[23698]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:08.166 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:07 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/4180388471' entity='client.admin' 2024-09-20T23:37:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:08 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1529313717' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-20T23:37:10.152 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:09 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1529313717' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-20T23:37:10.152 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:09 smithi137 ceph-mon[23698]: mgrmap e9: smithi137.eurecx(active, since 22s) 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "epoch": 9, 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "available": true, 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "active_name": "smithi137.eurecx", 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "num_standby": 0 2024-09-20T23:37:10.171 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:37:10.755 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for the mgr to restart... 2024-09-20T23:37:10.755 INFO:teuthology.orchestra.run.smithi137.stderr:Waiting for mgr epoch 9... 2024-09-20T23:37:11.046 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:10 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2293330542' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-20T23:37:16.104 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: { 2024-09-20T23:37:16.104 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-09-20T23:37:16.105 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: "initialized": true 2024-09-20T23:37:16.105 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: } 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: Active manager daemon smithi137.eurecx restarted 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: Activating manager daemon smithi137.eurecx 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: osdmap e3: 0 total, 0 up, 0 in 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: mgrmap e10: smithi137.eurecx(active, starting, since 0.0554129s) 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:37:16.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: [20/Sep/2024:23:37:15] ENGINE Bus STARTING 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: [20/Sep/2024:23:37:15] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: [20/Sep/2024:23:37:15] ENGINE Bus STARTED 2024-09-20T23:37:16.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:16 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:16.680 INFO:teuthology.orchestra.run.smithi137.stderr:mgr epoch 9 is available 2024-09-20T23:37:16.680 INFO:teuthology.orchestra.run.smithi137.stderr:Generating a dashboard self-signed certificate... 2024-09-20T23:37:17.278 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:17 smithi137 ceph-mon[23698]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-20T23:37:17.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:17 smithi137 ceph-mon[23698]: mgrmap e11: smithi137.eurecx(active, since 1.05747s) 2024-09-20T23:37:17.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:17 smithi137 ceph-mon[23698]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-20T23:37:17.279 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:17 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:17.419 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: Self-signed certificate created 2024-09-20T23:37:18.028 INFO:teuthology.orchestra.run.smithi137.stderr:Creating initial admin user... 2024-09-20T23:37:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:18 smithi137 ceph-mon[23698]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:18 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:18 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:18 smithi137 ceph-mon[23698]: mgrmap e12: smithi137.eurecx(active, since 2s) 2024-09-20T23:37:18.894 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$9n.PQl62xmdzVpQ3wSlt3O/DN49zlfnpvkAvp.2kmRZM2XnIuvpu2", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726875438, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-20T23:37:19.637 INFO:teuthology.orchestra.run.smithi137.stderr:Fetching dashboard port number... 2024-09-20T23:37:19.853 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:19 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:19.853 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:19 smithi137 ceph-mon[23698]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:19.853 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:19 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:20.729 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: 8443 2024-09-20T23:37:21.070 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:20 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1411066035' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-20T23:37:21.675 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: disabled 2024-09-20T23:37:21.682 INFO:teuthology.orchestra.run.smithi137.stderr:systemctl: inactive 2024-09-20T23:37:21.683 INFO:teuthology.orchestra.run.smithi137.stderr:firewalld.service is not enabled 2024-09-20T23:37:21.683 INFO:teuthology.orchestra.run.smithi137.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr:Ceph Dashboard is now available at: 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr: URL: https://smithi137.front.sepia.ceph.com:8443/ 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr: User: admin 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr: Password: ya9da7jj3u 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:21.686 INFO:teuthology.orchestra.run.smithi137.stderr:Enabling autotune for osd_memory_target 2024-09-20T23:37:23.086 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:22 smithi137 ceph-mon[23698]: mgrmap e13: smithi137.eurecx(active, since 6s) 2024-09-20T23:37:23.857 INFO:teuthology.orchestra.run.smithi137.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-09-20T23:37:24.411 INFO:teuthology.orchestra.run.smithi137.stderr:You can access the Ceph CLI with: 2024-09-20T23:37:24.411 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.411 INFO:teuthology.orchestra.run.smithi137.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:37:24.411 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr:Please consider enabling telemetry to help improve Ceph: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: ceph telemetry on 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr:For more information see: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-20T23:37:24.412 INFO:teuthology.orchestra.run.smithi137.stderr:Bootstrap complete. 2024-09-20T23:37:24.446 INFO:tasks.cephadm:Fetching config... 2024-09-20T23:37:24.446 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:37:24.446 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-20T23:37:24.463 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-20T23:37:24.463 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:37:24.463 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-20T23:37:24.518 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-20T23:37:24.519 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:37:24.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/keyring of=/dev/stdout 2024-09-20T23:37:24.583 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-20T23:37:24.584 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:37:24.584 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-20T23:37:24.639 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-20T23:37:24.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcyKknEYST9WHyVl2PDD88GRYVUB+/LZ8g4mDEdvjjmnnqsbkJ1KL2uHEqJpXy/ODpI8U9VoLz5zD4FK1xkJP8yKMYyhy5FmpjlR+0y4bjg3zOhLlHH7GXW6cCJLMECnQOMVrkBR0Khxrv80LpGw8QhSC05mQY+DVvf7HkPph8M9NHGbz4oKBzLedLSeAcB+UEhAt2gWfOMYe2JEV1CH8pGIp9Y4RWJii6b3HEBu7rc0YgbSkTXYvGxvbvzqfeRHBxoROyh2qQiPw5Gv13r3avpSG3IwDxPzAFdCfDX3tJE67wV/Pgq1oizF7day36DZJ6HsSCFH1rc5ikOSq6FBow3P5vRESH3WkM1w46zKcvrJUb9Mfy7niA4BwZLwzlu9CUK9/cPBHXe8v2CAEgdK1M1cgpOWK+RoTXXTGneIfm4kn3mMAs8ZqxUNtfP1xcdqtGheKs+9FBA29n+7PI+dbbRkbntjmuD7iFZbUcUPtulTK14FVQr0Rr+5qNjQFW+k= ceph-1150c2f6-77a9-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T23:37:24.712 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcyKknEYST9WHyVl2PDD88GRYVUB+/LZ8g4mDEdvjjmnnqsbkJ1KL2uHEqJpXy/ODpI8U9VoLz5zD4FK1xkJP8yKMYyhy5FmpjlR+0y4bjg3zOhLlHH7GXW6cCJLMECnQOMVrkBR0Khxrv80LpGw8QhSC05mQY+DVvf7HkPph8M9NHGbz4oKBzLedLSeAcB+UEhAt2gWfOMYe2JEV1CH8pGIp9Y4RWJii6b3HEBu7rc0YgbSkTXYvGxvbvzqfeRHBxoROyh2qQiPw5Gv13r3avpSG3IwDxPzAFdCfDX3tJE67wV/Pgq1oizF7day36DZJ6HsSCFH1rc5ikOSq6FBow3P5vRESH3WkM1w46zKcvrJUb9Mfy7niA4BwZLwzlu9CUK9/cPBHXe8v2CAEgdK1M1cgpOWK+RoTXXTGneIfm4kn3mMAs8ZqxUNtfP1xcdqtGheKs+9FBA29n+7PI+dbbRkbntjmuD7iFZbUcUPtulTK14FVQr0Rr+5qNjQFW+k= ceph-1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:37:24.723 DEBUG:teuthology.orchestra.run.smithi144:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcyKknEYST9WHyVl2PDD88GRYVUB+/LZ8g4mDEdvjjmnnqsbkJ1KL2uHEqJpXy/ODpI8U9VoLz5zD4FK1xkJP8yKMYyhy5FmpjlR+0y4bjg3zOhLlHH7GXW6cCJLMECnQOMVrkBR0Khxrv80LpGw8QhSC05mQY+DVvf7HkPph8M9NHGbz4oKBzLedLSeAcB+UEhAt2gWfOMYe2JEV1CH8pGIp9Y4RWJii6b3HEBu7rc0YgbSkTXYvGxvbvzqfeRHBxoROyh2qQiPw5Gv13r3avpSG3IwDxPzAFdCfDX3tJE67wV/Pgq1oizF7day36DZJ6HsSCFH1rc5ikOSq6FBow3P5vRESH3WkM1w46zKcvrJUb9Mfy7niA4BwZLwzlu9CUK9/cPBHXe8v2CAEgdK1M1cgpOWK+RoTXXTGneIfm4kn3mMAs8ZqxUNtfP1xcdqtGheKs+9FBA29n+7PI+dbbRkbntjmuD7iFZbUcUPtulTK14FVQr0Rr+5qNjQFW+k= ceph-1150c2f6-77a9-11ef-baf5-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-20T23:37:24.773 INFO:teuthology.orchestra.run.smithi144.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAcyKknEYST9WHyVl2PDD88GRYVUB+/LZ8g4mDEdvjjmnnqsbkJ1KL2uHEqJpXy/ODpI8U9VoLz5zD4FK1xkJP8yKMYyhy5FmpjlR+0y4bjg3zOhLlHH7GXW6cCJLMECnQOMVrkBR0Khxrv80LpGw8QhSC05mQY+DVvf7HkPph8M9NHGbz4oKBzLedLSeAcB+UEhAt2gWfOMYe2JEV1CH8pGIp9Y4RWJii6b3HEBu7rc0YgbSkTXYvGxvbvzqfeRHBxoROyh2qQiPw5Gv13r3avpSG3IwDxPzAFdCfDX3tJE67wV/Pgq1oizF7day36DZJ6HsSCFH1rc5ikOSq6FBow3P5vRESH3WkM1w46zKcvrJUb9Mfy7niA4BwZLwzlu9CUK9/cPBHXe8v2CAEgdK1M1cgpOWK+RoTXXTGneIfm4kn3mMAs8ZqxUNtfP1xcdqtGheKs+9FBA29n+7PI+dbbRkbntjmuD7iFZbUcUPtulTK14FVQr0Rr+5qNjQFW+k= ceph-1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:37:24.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-20T23:37:24.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:24 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3846638377' entity='client.admin' 2024-09-20T23:37:27.680 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-20T23:37:27.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-20T23:37:27.894 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:27 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2528100290' entity='client.admin' 2024-09-20T23:37:27.895 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:27 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:27.895 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:27 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:37:27.895 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:27 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:28.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:28 smithi137 ceph-mon[23698]: Deploying daemon alertmanager.smithi137 on smithi137 2024-09-20T23:37:31.219 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi144 2024-09-20T23:37:31.219 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:37:31.219 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.conf 2024-09-20T23:37:31.238 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:37:31.238 DEBUG:teuthology.orchestra.run.smithi144:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:37:31.296 INFO:tasks.cephadm:Adding host smithi144 to orchestrator... 2024-09-20T23:37:31.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch host add smithi144 2024-09-20T23:37:31.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:31 smithi137 ceph-mon[23698]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:31.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:31 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:34 smithi137 ceph-mon[23698]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi144", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:35.706 INFO:teuthology.orchestra.run.smithi137.stdout:Added host 'smithi144' with addr '172.21.15.144' 2024-09-20T23:37:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:36 smithi137 ceph-mon[23698]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:36 smithi137 ceph-mon[23698]: Deploying cephadm binary to smithi144 2024-09-20T23:37:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:36 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:36 smithi137 ceph-mon[23698]: Added host smithi144 2024-09-20T23:37:37.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch host ls --format=json 2024-09-20T23:37:38.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:38 smithi137 ceph-mon[23698]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:38.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:38 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:38.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:38 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:37:38.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:38 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-20T23:37:38.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:38 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:37:39.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:39 smithi137 ceph-mon[23698]: Deploying daemon crash.smithi137 on smithi137 2024-09-20T23:37:39.968 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:37:39.968 INFO:teuthology.orchestra.run.smithi137.stdout:[{"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}, {"addr": "172.21.15.144", "hostname": "smithi144", "labels": [], "status": ""}] 2024-09-20T23:37:40.645 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-20T23:37:40.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd crush tunables default 2024-09-20T23:37:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:40 smithi137 ceph-mon[23698]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:40 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:41.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:41 smithi137 ceph-mon[23698]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:37:42.854 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:42 smithi137 ceph-mon[23698]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:43.721 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2643729732' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:37:43.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:43 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:44.346 INFO:tasks.cephadm:Adding mon.smithi137 on smithi137 2024-09-20T23:37:44.347 INFO:tasks.cephadm:Adding mon.smithi144 on smithi144 2024-09-20T23:37:44.347 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch apply mon '2;smithi137:172.21.15.137=smithi137;smithi144:172.21.15.144=smithi144' 2024-09-20T23:37:45.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:44 smithi137 ceph-mon[23698]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:45.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:44 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:37:45.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:44 smithi137 ceph-mon[23698]: Deploying daemon grafana.smithi137 on smithi137 2024-09-20T23:37:45.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:44 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2643729732' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-20T23:37:45.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:44 smithi137 ceph-mon[23698]: osdmap e4: 0 total, 0 up, 0 in 2024-09-20T23:37:46.060 INFO:teuthology.orchestra.run.smithi144.stdout:Scheduled mon update... 2024-09-20T23:37:46.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:46 smithi137 ceph-mon[23698]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:46.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:46 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:46.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:46 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:46.616 DEBUG:teuthology.orchestra.run.smithi144:mon.smithi144> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service 2024-09-20T23:37:46.619 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:37:46.619 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:37:47.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:47 smithi137 ceph-mon[23698]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi137:172.21.15.137=smithi137;smithi144:172.21.15.144=smithi144", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:37:47.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:47 smithi137 ceph-mon[23698]: Saving service mon spec with placement smithi137:172.21.15.137=smithi137;smithi144:172.21.15.144=smithi144;count:2 2024-09-20T23:37:48.506 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:37:48.506 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:37:48.508 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:37:48.642 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:48 smithi137 ceph-mon[23698]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:49.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:49 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1375640571' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:37:50.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:37:50.016 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:37:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:50 smithi137 ceph-mon[23698]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:51.888 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:37:51.888 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:37:51.890 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:37:52.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:52 smithi137 ceph-mon[23698]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:52.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:52 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1749984730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:37:53.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:37:53.561 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:37:54.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:54 smithi137 ceph-mon[23698]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:55.291 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:37:55.291 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:37:55.293 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:37:55.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:55 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/28249454' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:37:56.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:56 smithi137 ceph-mon[23698]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:56.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:56 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:37:56.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:37:56.847 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:37:57.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:57 smithi137 ceph-mon[23698]: Deploying daemon node-exporter.smithi137 on smithi137 2024-09-20T23:37:58.665 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:37:58.665 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:37:58.667 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:37:58.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:58 smithi137 ceph-mon[23698]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:37:59.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:37:59 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1808570369' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:00.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:00.232 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:00.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:00 smithi137 ceph-mon[23698]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:00.872 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:00 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:02.016 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:02.016 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:02.018 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:02.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:02 smithi137 ceph-mon[23698]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:02.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:02 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1866420142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:03.594 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:03.594 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:04.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:04 smithi137 ceph-mon[23698]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:04.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:04 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:04.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:04 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-20T23:38:05.399 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:05.400 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:05.402 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:06.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:06 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:06.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:06 smithi137 ceph-mon[23698]: from='mgr.14160 172.21.15.137:0/1859018236' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-20T23:38:06.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:06 smithi137 ceph-mon[23698]: mgrmap e14: smithi137.eurecx(active, since 50s) 2024-09-20T23:38:06.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:06 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/542969301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:06.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:06.982 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:08.706 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:08.706 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:08.708 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:08.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:08 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/2971348730' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:10.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:10.223 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:11.550 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:11 smithi137 ceph-mon[23698]: Active manager daemon smithi137.eurecx restarted 2024-09-20T23:38:11.550 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:11 smithi137 ceph-mon[23698]: Activating manager daemon smithi137.eurecx 2024-09-20T23:38:11.550 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:11 smithi137 ceph-mon[23698]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T23:38:12.117 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:12.118 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:12.120 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: mgrmap e15: smithi137.eurecx(active, starting, since 0.0534393s) 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:38:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: [20/Sep/2024:23:38:11] ENGINE Bus STARTING 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: [20/Sep/2024:23:38:11] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: [20/Sep/2024:23:38:11] ENGINE Bus STARTED 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/2596187639' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:12.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:13.671 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:13 smithi137 ceph-mon[23698]: mgrmap e16: smithi137.eurecx(active, since 1.05851s) 2024-09-20T23:38:13.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:13.837 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:13.837 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:15.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:15 smithi137 ceph-mon[23698]: mgrmap e17: smithi137.eurecx(active, since 2s) 2024-09-20T23:38:15.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:15 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:15.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:15 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:15.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:15 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:38:16.412 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:16 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:38:16.412 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:16 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:38:16.412 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:16.546 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:16.546 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:16.548 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:17.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1366836855' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:17.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd/host:smithi144", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-09-20T23:38:17.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:18.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:18.288 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:18.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:18 smithi137 ceph-mon[23698]: Deploying daemon crash.smithi144 on smithi144 2024-09-20T23:38:20.727 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:20.727 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:20.729 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:21.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:21.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:38:21.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-20T23:38:21.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:38:21.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:21.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: Deploying daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:38:21.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/733723948' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:21.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:22.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:22.345 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:24.293 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:24.293 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:24.295 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:24 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/3305816665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:25.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:25.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:38:25.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:25.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:25 smithi137 ceph-mon[23698]: Deploying daemon mon.smithi144 on smithi144 2024-09-20T23:38:26.193 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:26.193 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:29.231 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:29.231 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":1,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:36:10.055803Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-20T23:38:29.233 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 1 2024-09-20T23:38:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:29 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/2990218059' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:30.136 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:30 smithi144 systemd[1]: Started Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:38:30.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-09-20T23:38:30.898 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mon dump -f json 2024-09-20T23:38:35.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:38:35.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: mon.smithi137 calling monitor election 2024-09-20T23:38:35.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: mon.smithi144 calling monitor election 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: monmap e2: 2 mons at {smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],smithi144=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: fsmap 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: mgrmap e17: smithi137.eurecx(active, since 23s) 2024-09-20T23:38:35.386 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:38:35.387 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:38:35.387 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: overall HEALTH_OK 2024-09-20T23:38:35.387 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:38:35.387 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:38:35.387 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:38:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: mon.smithi137 calling monitor election 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: mon.smithi144 calling monitor election 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: monmap e2: 2 mons at {smithi137=[v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0],smithi144=[v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0]} 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: fsmap 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: osdmap e5: 0 total, 0 up, 0 in 2024-09-20T23:38:35.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: mgrmap e17: smithi137.eurecx(active, since 23s) 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: overall HEALTH_OK 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/4092040592' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:38:35.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:35.846 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:38:35.847 INFO:teuthology.orchestra.run.smithi144.stdout:{"epoch":2,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","modified":"2024-09-20T23:38:30.082097Z","created":"2024-09-20T23:36:10.055803Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi137","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:3300","nonce":0},{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi144","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:3300","nonce":0},{"type":"v1","addr":"172.21.15.144:6789","nonce":0}]},"addr":"172.21.15.144:6789/0","public_addr":"172.21.15.144:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-20T23:38:35.848 INFO:teuthology.orchestra.run.smithi144.stderr:dumped monmap epoch 2 2024-09-20T23:38:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:36 smithi137 ceph-mon[23698]: Deploying daemon node-exporter.smithi144 on smithi144 2024-09-20T23:38:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:36 smithi137 ceph-mon[23698]: mgrmap e18: smithi137.eurecx(active, since 23s), standbys: smithi144.ygsngz 2024-09-20T23:38:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:36 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:38:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:36 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1373000284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:36.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:36 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:36 smithi144 ceph-mon[26057]: Deploying daemon node-exporter.smithi144 on smithi144 2024-09-20T23:38:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:36 smithi144 ceph-mon[26057]: mgrmap e18: smithi137.eurecx(active, since 23s), standbys: smithi144.ygsngz 2024-09-20T23:38:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:36 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:38:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:36 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/1373000284' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-20T23:38:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:36 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:38:36.483 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-20T23:38:36.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph config generate-minimal-conf 2024-09-20T23:38:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:37 smithi137 ceph-mon[23698]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:37.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:37 smithi144 ceph-mon[26057]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:37.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:38.326 INFO:teuthology.orchestra.run.smithi137.stdout:# minimal ceph.conf for 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:38:38.326 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-09-20T23:38:38.326 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:38:38.326 INFO:teuthology.orchestra.run.smithi137.stdout: mon_host = [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] 2024-09-20T23:38:38.918 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-20T23:38:38.918 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:38:38.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T23:38:38.945 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:38:38.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:38:39.010 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:38:39.010 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.conf 2024-09-20T23:38:39.042 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:38:39.043 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:38:39.111 INFO:tasks.cephadm:Deploying OSDs... 2024-09-20T23:38:39.111 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:38:39.112 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T23:38:39.128 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T23:38:39.128 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:39.184 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-20 23:37:56.590569959 +0000 2024-09-20T23:38:39.185 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-20 23:35:28.530282995 +0000 2024-09-20T23:38:39.185 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-20 23:35:28.530282995 +0000 2024-09-20T23:38:39.185 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-20 23:35:28.530282995 +0000 2024-09-20T23:38:39.185 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T23:38:39.247 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-20T23:38:39.248 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-20T23:38:39.248 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000196863 s, 2.6 MB/s 2024-09-20T23:38:39.249 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T23:38:39.305 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-09-20T23:38:39.360 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-20 23:37:56.591569941 +0000 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-20 23:35:28.877276827 +0000 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-20 23:35:28.877276827 +0000 2024-09-20T23:38:39.361 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-20 23:35:28.877276827 +0000 2024-09-20T23:38:39.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T23:38:39.422 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:39 smithi137 ceph-mon[23698]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:39.423 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:39 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2542623393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:39.423 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:39.425 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-20T23:38:39.425 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-20T23:38:39.426 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000194498 s, 2.6 MB/s 2024-09-20T23:38:39.427 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T23:38:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:39 smithi144 ceph-mon[26057]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:39 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2542623393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:39.484 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-09-20T23:38:39.540 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T23:38:39.540 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:39.540 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-09-20T23:38:39.540 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:39.541 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:39.541 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-20 23:37:56.591569941 +0000 2024-09-20T23:38:39.541 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-20 23:35:29.191271246 +0000 2024-09-20T23:38:39.541 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-20 23:35:29.191271246 +0000 2024-09-20T23:38:39.541 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-20 23:35:29.191271246 +0000 2024-09-20T23:38:39.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T23:38:39.603 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-20T23:38:39.603 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-20T23:38:39.603 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000312758 s, 1.6 MB/s 2024-09-20T23:38:39.605 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T23:38:39.661 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:39.716 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-20 23:37:56.592569924 +0000 2024-09-20T23:38:39.717 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-20 23:35:29.520265398 +0000 2024-09-20T23:38:39.717 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-20 23:35:29.520265398 +0000 2024-09-20T23:38:39.717 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-20 23:35:29.520265398 +0000 2024-09-20T23:38:39.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T23:38:39.779 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-20T23:38:39.779 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-20T23:38:39.779 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000194843 s, 2.6 MB/s 2024-09-20T23:38:39.781 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T23:38:39.837 DEBUG:teuthology.orchestra.run.smithi144:> set -ex 2024-09-20T23:38:39.837 DEBUG:teuthology.orchestra.run.smithi144:> dd if=/scratch_devs of=/dev/stdout 2024-09-20T23:38:39.857 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-20T23:38:39.857 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_1 2024-09-20T23:38:39.915 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-20T23:38:39.915 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-09-20T23:38:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:39.915 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:39.916 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-20 23:38:35.678466513 +0000 2024-09-20T23:38:39.916 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-20 23:35:31.328528187 +0000 2024-09-20T23:38:39.916 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-20 23:35:31.328528187 +0000 2024-09-20T23:38:39.916 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-20 23:35:31.328528187 +0000 2024-09-20T23:38:39.916 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-20T23:38:39.982 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-20T23:38:39.982 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-20T23:38:39.982 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000293672 s, 1.7 MB/s 2024-09-20T23:38:39.983 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-20T23:38:40.042 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_2 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-20 23:38:35.678466513 +0000 2024-09-20T23:38:40.099 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-20 23:35:31.654522918 +0000 2024-09-20T23:38:40.100 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-20 23:35:31.654522918 +0000 2024-09-20T23:38:40.100 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-20 23:35:31.654522918 +0000 2024-09-20T23:38:40.100 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-20T23:38:40.165 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-20T23:38:40.165 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-20T23:38:40.165 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000213258 s, 2.4 MB/s 2024-09-20T23:38:40.167 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-20T23:38:40.225 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_3 2024-09-20T23:38:40.281 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-20T23:38:40.281 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-20 23:38:35.678466513 +0000 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-20 23:35:31.970517811 +0000 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-20 23:35:31.970517811 +0000 2024-09-20T23:38:40.282 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-20 23:35:31.970517811 +0000 2024-09-20T23:38:40.282 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-20T23:38:40.344 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:40 smithi144 ceph-mon[26057]: Deploying daemon prometheus.smithi137 on smithi137 2024-09-20T23:38:40.346 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-20T23:38:40.346 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-20T23:38:40.346 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000243915 s, 2.1 MB/s 2024-09-20T23:38:40.348 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-20T23:38:40.405 DEBUG:teuthology.orchestra.run.smithi144:> stat /dev/vg_nvme/lv_4 2024-09-20T23:38:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:40 smithi137 ceph-mon[23698]: Deploying daemon prometheus.smithi137 on smithi137 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Context: system_u:object_r:device_t:s0 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Access: 2024-09-20 23:38:35.679466496 +0000 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Modify: 2024-09-20 23:35:32.289512655 +0000 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout:Change: 2024-09-20 23:35:32.289512655 +0000 2024-09-20T23:38:40.461 INFO:teuthology.orchestra.run.smithi144.stdout: Birth: 2024-09-20 23:35:32.289512655 +0000 2024-09-20T23:38:40.462 DEBUG:teuthology.orchestra.run.smithi144:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-20T23:38:40.525 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records in 2024-09-20T23:38:40.525 INFO:teuthology.orchestra.run.smithi144.stderr:1+0 records out 2024-09-20T23:38:40.525 INFO:teuthology.orchestra.run.smithi144.stderr:512 bytes copied, 0.000195139 s, 2.6 MB/s 2024-09-20T23:38:40.526 DEBUG:teuthology.orchestra.run.smithi144:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-20T23:38:40.582 INFO:tasks.cephadm:Deploying osd.0 on smithi137 with /dev/vg_nvme/lv_4... 2024-09-20T23:38:40.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T23:38:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:41 smithi137 ceph-mon[23698]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:41 smithi144 ceph-mon[26057]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:42 smithi137 ceph-mon[23698]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:42 smithi144 ceph-mon[26057]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:44.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:44 smithi137 ceph-mon[23698]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:44.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:44 smithi144 ceph-mon[26057]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:46.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:46 smithi137 ceph-mon[23698]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:46 smithi144 ceph-mon[26057]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:48.595 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:48 smithi137 ceph-mon[23698]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:48 smithi144 ceph-mon[26057]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:51.063 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:50 smithi144 ceph-mon[26057]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:51.063 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:51.063 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:38:51.063 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:51.063 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:38:51.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:50 smithi137 ceph-mon[23698]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:51.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:51.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:38:51.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:51.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:38:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:38:51.482 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-09-20T23:38:52.592 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:52.592 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:38:52.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:52.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:52.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:38:52.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:52 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:38:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:52.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:52.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:38:52.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:52 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:54.840 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:54 smithi137 ceph-mon[23698]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:54 smithi144 ceph-mon[26057]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:55.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:55 smithi137 ceph-mon[23698]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:55.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:38:55.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:38:55.758 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:55.844 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:55 smithi144 ceph-mon[26057]: from='client.14238 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:38:55.845 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:38:55.845 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:38:55.845 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:56.855 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: Reconfiguring mon.smithi137 (unknown last config time)... 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:56.856 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:56 smithi137 ceph-mon[23698]: Reconfiguring daemon mon.smithi137 on smithi137 2024-09-20T23:38:56.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: Reconfiguring mon.smithi137 (unknown last config time)... 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:56.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:56 smithi144 ceph-mon[26057]: Reconfiguring daemon mon.smithi137 on smithi137 2024-09-20T23:38:59.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:59.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:59.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: Reconfiguring mgr.smithi137.eurecx (unknown last config time)... 2024-09-20T23:38:59.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:38:59.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:38:59.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:59.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:38:58 smithi137 ceph-mon[23698]: Reconfiguring daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: Reconfiguring mgr.smithi137.eurecx (unknown last config time)... 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:38:59.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:38:58 smithi144 ceph-mon[26057]: Reconfiguring daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:39:01.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:01.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3548412878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfd14532-0384-4863-ae9b-eaf2bc30783f"}]: dispatch 2024-09-20T23:39:01.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3548412878' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfd14532-0384-4863-ae9b-eaf2bc30783f"}]': finished 2024-09-20T23:39:01.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T23:39:01.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:01.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:00 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/264922085' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:39:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3548412878' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cfd14532-0384-4863-ae9b-eaf2bc30783f"}]: dispatch 2024-09-20T23:39:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3548412878' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cfd14532-0384-4863-ae9b-eaf2bc30783f"}]': finished 2024-09-20T23:39:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: osdmap e6: 1 total, 0 up, 1 in 2024-09-20T23:39:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:00 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/264922085' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:39:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:02 smithi137 ceph-mon[23698]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:02 smithi144 ceph-mon[26057]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:05.177 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:04 smithi137 ceph-mon[23698]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:05.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:04 smithi144 ceph-mon[26057]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:06.540 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:06 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:06.540 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:06 smithi144 ceph-mon[26057]: Reconfiguring alertmanager.smithi137 (dependencies changed)... 2024-09-20T23:39:06.540 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:06 smithi144 ceph-mon[26057]: Reconfiguring daemon alertmanager.smithi137 on smithi137 2024-09-20T23:39:06.581 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:06 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:06.581 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:06 smithi137 ceph-mon[23698]: Reconfiguring alertmanager.smithi137 (dependencies changed)... 2024-09-20T23:39:06.581 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:06 smithi137 ceph-mon[23698]: Reconfiguring daemon alertmanager.smithi137 on smithi137 2024-09-20T23:39:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:07 smithi144 ceph-mon[26057]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:39:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:07.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:07 smithi137 ceph-mon[23698]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:07.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:39:07.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:08.523 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:08 smithi137 ceph-mon[23698]: Deploying daemon osd.0 on smithi137 2024-09-20T23:39:08.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:08 smithi144 ceph-mon[26057]: Deploying daemon osd.0 on smithi137 2024-09-20T23:39:09.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:09 smithi137 ceph-mon[23698]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:09 smithi144 ceph-mon[26057]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:10.557 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:10 smithi137 ceph-mon[23698]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:10.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:10 smithi144 ceph-mon[26057]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:11.527 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:39:11.527 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:39:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:39:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:39:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: Reconfiguring crash.smithi137 (monmap changed)... 2024-09-20T23:39:12.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:39:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:12.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:12 smithi137 ceph-mon[23698]: Reconfiguring daemon crash.smithi137 on smithi137 2024-09-20T23:39:12.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:12.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:12.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: Reconfiguring crash.smithi137 (monmap changed)... 2024-09-20T23:39:12.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:39:12.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:12.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:12 smithi144 ceph-mon[26057]: Reconfiguring daemon crash.smithi137 on smithi137 2024-09-20T23:39:14.644 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:14 smithi137 ceph-mon[23698]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:14 smithi144 ceph-mon[26057]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:16.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:16 smithi137 ceph-mon[23698]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:16.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:16 smithi144 ceph-mon[26057]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:16 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:18.884 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:18 smithi137 ceph-mon[23698]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:18.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:18 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:18.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:18 smithi137 ceph-mon[23698]: Reconfiguring grafana.smithi137 (dependencies changed)... 2024-09-20T23:39:18.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:18 smithi137 ceph-mon[23698]: Reconfiguring daemon grafana.smithi137 on smithi137 2024-09-20T23:39:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:18 smithi144 ceph-mon[26057]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:18 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:18 smithi144 ceph-mon[26057]: Reconfiguring grafana.smithi137 (dependencies changed)... 2024-09-20T23:39:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:18 smithi144 ceph-mon[26057]: Reconfiguring daemon grafana.smithi137 on smithi137 2024-09-20T23:39:19.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:19 smithi137 ceph-mon[23698]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T23:39:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:19 smithi144 ceph-mon[26057]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T23:39:20.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:20 smithi137 ceph-mon[23698]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:20.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:20 smithi137 ceph-mon[23698]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T23:39:20.886 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:20 smithi137 ceph-mon[23698]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T23:39:20.886 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:20 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:20.886 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:20 smithi137 ceph-mon[23698]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:39:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:20 smithi144 ceph-mon[26057]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:20 smithi144 ceph-mon[26057]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T23:39:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:20 smithi144 ceph-mon[26057]: osdmap e7: 1 total, 0 up, 1 in 2024-09-20T23:39:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:20 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:20 smithi144 ceph-mon[26057]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:39:21.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:21 smithi137 ceph-mon[23698]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:39:21.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:21 smithi137 ceph-mon[23698]: osdmap e8: 1 total, 0 up, 1 in 2024-09-20T23:39:21.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:21.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.120 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:21 smithi144 ceph-mon[26057]: from='osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:39:22.120 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:21 smithi144 ceph-mon[26057]: osdmap e8: 1 total, 0 up, 1 in 2024-09-20T23:39:22.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.832 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:39:22.832 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545] boot 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: Reconfiguring crash.smithi144 (monmap changed)... 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:22.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:22 smithi144 ceph-mon[26057]: Reconfiguring daemon crash.smithi144 on smithi144 2024-09-20T23:39:22.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:39:22.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:39:22.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: osd.0 [v2:172.21.15.137:6802/3729731545,v1:172.21.15.137:6803/3729731545] boot 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: osdmap e9: 1 total, 1 up, 1 in 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: Reconfiguring crash.smithi144 (monmap changed)... 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:22.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:22 smithi137 ceph-mon[23698]: Reconfiguring daemon crash.smithi144 on smithi144 2024-09-20T23:39:23.627 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 0 on host 'smithi137' 2024-09-20T23:39:24.281 DEBUG:teuthology.orchestra.run.smithi137:osd.0> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.0.service 2024-09-20T23:39:24.283 INFO:tasks.cephadm:Deploying osd.1 on smithi137 with /dev/vg_nvme/lv_3... 2024-09-20T23:39:24.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T23:39:24.623 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:24.623 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: osdmap e10: 1 total, 1 up, 1 in 2024-09-20T23:39:24.623 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:24.623 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: Reconfiguring mgr.smithi144.ygsngz (monmap changed)... 2024-09-20T23:39:24.623 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:39:24.624 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:39:24.624 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:24.624 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: Reconfiguring daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:39:24.624 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:24 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: osdmap e10: 1 total, 1 up, 1 in 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: Reconfiguring mgr.smithi144.ygsngz (monmap changed)... 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:39:24.773 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:24.774 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: Reconfiguring daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:39:24.774 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:24 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: Reconfiguring mon.smithi144 (monmap changed)... 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:26.025 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:25 smithi137 ceph-mon[23698]: Reconfiguring daemon mon.smithi144 on smithi144 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: Reconfiguring mon.smithi144 (monmap changed)... 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:26.089 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:25 smithi144 ceph-mon[26057]: Reconfiguring daemon mon.smithi144 on smithi144 2024-09-20T23:39:26.808 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:39:26.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-09-20T23:39:27.420 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:27.420 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.420 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:39:27.420 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:39:27.420 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.137:9093"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.137:9093"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.137:3000"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.137:3000"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.137:9095"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.137:9095"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:27.421 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:27 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:39:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.137:9093"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.137:9093"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.137:3000"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.137:3000"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.137:9095"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.137:9095"}]: dispatch 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:27.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:27.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:27.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:27 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:28.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:28 smithi137 ceph-mon[23698]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:28.999 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:28 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:29.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:28 smithi144 ceph-mon[26057]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:29.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:28 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:30.149 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:29 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:39:30.149 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:29 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:39:30.149 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:29 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:30.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:29 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:39:30.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:29 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:39:30.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:29 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:31.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:30 smithi137 ceph-mon[23698]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:30 smithi137 ceph-mon[23698]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:30 smithi144 ceph-mon[26057]: from='client.14256 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:30 smithi144 ceph-mon[26057]: pgmap v37: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:33.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:32 smithi137 ceph-mon[23698]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:32 smithi144 ceph-mon[26057]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1668465352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad3fdbc-67f4-4f11-9532-64cf3fb7910d"}]: dispatch 2024-09-20T23:39:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1668465352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cad3fdbc-67f4-4f11-9532-64cf3fb7910d"}]': finished 2024-09-20T23:39:34.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T23:39:34.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:34.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:34 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2363732797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:39:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1668465352' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cad3fdbc-67f4-4f11-9532-64cf3fb7910d"}]: dispatch 2024-09-20T23:39:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1668465352' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cad3fdbc-67f4-4f11-9532-64cf3fb7910d"}]': finished 2024-09-20T23:39:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: osdmap e11: 2 total, 1 up, 2 in 2024-09-20T23:39:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:34 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2363732797' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:39:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:35 smithi137 ceph-mon[23698]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:35 smithi144 ceph-mon[26057]: pgmap v40: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:37.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:37 smithi137 ceph-mon[23698]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:37 smithi144 ceph-mon[26057]: pgmap v41: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:39.251 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:39.251 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: pgmap v42: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:39.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:40.483 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:40 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi137 2024-09-20T23:39:40.483 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:40 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi137 to 12864M 2024-09-20T23:39:40.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:40 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi137 2024-09-20T23:39:40.780 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:40 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi137 to 12864M 2024-09-20T23:39:41.362 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:41 smithi137 ceph-mon[23698]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:41.362 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:41 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:41.362 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:41 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:39:41.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:41 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:41 smithi144 ceph-mon[26057]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:41 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:41 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:39:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:41 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:42.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:42 smithi137 ceph-mon[23698]: Deploying daemon osd.1 on smithi137 2024-09-20T23:39:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:42 smithi144 ceph-mon[26057]: Deploying daemon osd.1 on smithi137 2024-09-20T23:39:43.497 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:43 smithi137 ceph-mon[23698]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:43 smithi144 ceph-mon[26057]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:45 smithi137 ceph-mon[23698]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:45 smithi144 ceph-mon[26057]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:47.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:46 smithi137 ceph-mon[23698]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:47.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:47.125 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:46 smithi144 ceph-mon[26057]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:49.001 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:48 smithi137 ceph-mon[23698]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:48 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:48 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:48 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:49.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:48 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:49.020 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:48 smithi144 ceph-mon[26057]: pgmap v47: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:49.021 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:48 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:49.021 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:48 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:39:49.021 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:48 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:49.021 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:48 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:39:50.415 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 1 on host 'smithi137' 2024-09-20T23:39:50.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:50 smithi137 ceph-mon[23698]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:50.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:50.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:50 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:50 smithi144 ceph-mon[26057]: pgmap v48: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:50 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:51.363 DEBUG:teuthology.orchestra.run.smithi137:osd.1> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.1.service 2024-09-20T23:39:51.365 INFO:tasks.cephadm:Deploying osd.2 on smithi137 with /dev/vg_nvme/lv_2... 2024-09-20T23:39:51.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T23:39:51.651 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:51 smithi137 ceph-mon[23698]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T23:39:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:51 smithi144 ceph-mon[26057]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T23:39:52.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-mon[23698]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:52.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-mon[23698]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T23:39:52.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-mon[23698]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T23:39:52.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:52.505 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-mon[23698]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:39:52.505 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[38146]: 2024-09-20T23:39:52.428+0000 7f7b6c954700 -1 osd.1 0 waiting for initial osdmap 2024-09-20T23:39:52.505 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:39:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[38146]: 2024-09-20T23:39:52.430+0000 7f7b652e7700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:39:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:52 smithi144 ceph-mon[26057]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:52 smithi144 ceph-mon[26057]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T23:39:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:52 smithi144 ceph-mon[26057]: osdmap e12: 2 total, 1 up, 2 in 2024-09-20T23:39:52.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:52 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:52.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:52 smithi144 ceph-mon[26057]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:39:53.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:53 smithi137 ceph-mon[23698]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:39:53.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:53 smithi137 ceph-mon[23698]: osdmap e13: 2 total, 1 up, 2 in 2024-09-20T23:39:53.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:53.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:53 smithi144 ceph-mon[26057]: from='osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:39:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:53 smithi144 ceph-mon[26057]: osdmap e13: 2 total, 1 up, 2 in 2024-09-20T23:39:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:54.533 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:39:54.534 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:39:54.534 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:54.534 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046] boot 2024-09-20T23:39:54.534 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T23:39:54.534 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:54 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:39:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:39:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: pgmap v52: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-09-20T23:39:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: osd.1 [v2:172.21.15.137:6810/1601068046,v1:172.21.15.137:6811/1601068046] boot 2024-09-20T23:39:54.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: osdmap e14: 2 total, 2 up, 2 in 2024-09-20T23:39:54.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:54 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:39:55.394 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:39:55.411 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-09-20T23:39:55.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:55 smithi137 ceph-mon[23698]: osdmap e15: 2 total, 2 up, 2 in 2024-09-20T23:39:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:55 smithi144 ceph-mon[26057]: osdmap e15: 2 total, 2 up, 2 in 2024-09-20T23:39:57.340 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:57 smithi137 ceph-mon[23698]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:39:57.340 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:57 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:57.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:57 smithi144 ceph-mon[26057]: pgmap v55: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:39:57.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:57 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:39:59.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:59 smithi137 ceph-mon[23698]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:39:59.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:59 smithi137 ceph-mon[23698]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:59.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:39:59.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:39:59.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:39:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:39:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:59 smithi144 ceph-mon[26057]: pgmap v56: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:39:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:59 smithi144 ceph-mon[26057]: from='client.14274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:39:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:39:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:39:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:39:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:00.397 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:00 smithi137 ceph-mon[23698]: overall HEALTH_OK 2024-09-20T23:40:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:00 smithi144 ceph-mon[26057]: overall HEALTH_OK 2024-09-20T23:40:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:01 smithi137 ceph-mon[23698]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:01.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:01 smithi144 ceph-mon[26057]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:02 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi137 to 6432M 2024-09-20T23:40:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:02 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi137 to 6432M 2024-09-20T23:40:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:03.537 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:03.537 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/779126494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc60058a-b022-4ab7-b855-6341004902cd"}]: dispatch 2024-09-20T23:40:03.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/779126494' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc60058a-b022-4ab7-b855-6341004902cd"}]': finished 2024-09-20T23:40:03.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T23:40:03.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:03.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:03 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1567187886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/779126494' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bc60058a-b022-4ab7-b855-6341004902cd"}]: dispatch 2024-09-20T23:40:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/779126494' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bc60058a-b022-4ab7-b855-6341004902cd"}]': finished 2024-09-20T23:40:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: osdmap e16: 3 total, 2 up, 3 in 2024-09-20T23:40:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:03.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:03 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1567187886' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:04.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:04 smithi137 ceph-mon[23698]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:04.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:04 smithi144 ceph-mon[26057]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:07.330 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:07 smithi137 ceph-mon[23698]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:07.330 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:07.330 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:07 smithi144 ceph-mon[26057]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:07.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:09.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:09 smithi137 ceph-mon[23698]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:09.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:09 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:40:09.361 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:09 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:09.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:09 smithi144 ceph-mon[26057]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:09.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:09 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:40:09.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:09 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:10.159 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:10 smithi137 ceph-mon[23698]: Deploying daemon osd.2 on smithi137 2024-09-20T23:40:10.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:10 smithi144 ceph-mon[26057]: Deploying daemon osd.2 on smithi137 2024-09-20T23:40:11.357 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:11 smithi137 ceph-mon[23698]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:11.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:11 smithi144 ceph-mon[26057]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:12.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:40:12.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:40:12.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:40:12.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:40:13.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:13 smithi137 ceph-mon[23698]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:13.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:13 smithi144 ceph-mon[26057]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:15.157 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:14 smithi137 ceph-mon[23698]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:15.157 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:15.157 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:15.157 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:15.158 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:14 smithi144 ceph-mon[26057]: pgmap v65: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:17.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:16 smithi137 ceph-mon[23698]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:17.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:16 smithi144 ceph-mon[26057]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:17.636 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 2 on host 'smithi137' 2024-09-20T23:40:18.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:17 smithi137 ceph-mon[23698]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:40:18.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:17 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:17 smithi144 ceph-mon[26057]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:40:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:17 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:18.239 DEBUG:teuthology.orchestra.run.smithi137:osd.2> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.2.service 2024-09-20T23:40:18.242 INFO:tasks.cephadm:Deploying osd.3 on smithi137 with /dev/vg_nvme/lv_1... 2024-09-20T23:40:18.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T23:40:19.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-mon[23698]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:19.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-mon[23698]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T23:40:19.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-mon[23698]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T23:40:19.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:19.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-mon[23698]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:40:19.179 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[41818]: 2024-09-20T23:40:18.908+0000 7fe4d2cc1700 -1 osd.2 0 waiting for initial osdmap 2024-09-20T23:40:19.179 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[41818]: 2024-09-20T23:40:18.912+0000 7fe4cce57700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:40:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:18 smithi144 ceph-mon[26057]: pgmap v67: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:18 smithi144 ceph-mon[26057]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T23:40:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:18 smithi144 ceph-mon[26057]: osdmap e17: 3 total, 2 up, 3 in 2024-09-20T23:40:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:18 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:18 smithi144 ceph-mon[26057]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:40:20.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:19 smithi137 ceph-mon[23698]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:40:20.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:19 smithi137 ceph-mon[23698]: osdmap e18: 3 total, 2 up, 3 in 2024-09-20T23:40:20.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:19 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:20.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:19 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:19 smithi144 ceph-mon[26057]: from='osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:40:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:19 smithi144 ceph-mon[26057]: osdmap e18: 3 total, 2 up, 3 in 2024-09-20T23:40:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:19 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:19 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:21.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:40:21.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:40:21.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:21.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:21.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294] boot 2024-09-20T23:40:21.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T23:40:21.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:20 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:40:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:40:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: pgmap v70: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-09-20T23:40:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: osd.2 [v2:172.21.15.137:6818/3517536294,v1:172.21.15.137:6819/3517536294] boot 2024-09-20T23:40:21.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: osdmap e19: 3 total, 3 up, 3 in 2024-09-20T23:40:21.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:20 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:40:22.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:21 smithi137 ceph-mon[23698]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T23:40:22.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-20T23:40:22.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:22.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:21 smithi144 ceph-mon[26057]: osdmap e20: 3 total, 3 up, 3 in 2024-09-20T23:40:22.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-09-20T23:40:22.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:40:22.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_1 2024-09-20T23:40:23.076 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:22 smithi137 ceph-mon[23698]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:23.077 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-20T23:40:23.077 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:22 smithi137 ceph-mon[23698]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T23:40:23.077 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T23:40:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:22 smithi144 ceph-mon[26057]: pgmap v73: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-09-20T23:40:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:22 smithi144 ceph-mon[26057]: osdmap e21: 3 total, 3 up, 3 in 2024-09-20T23:40:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-20T23:40:23.671 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43272]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T23:40:23.671 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43272]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:23.672 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43272]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:23.924 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43306]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T23:40:23.925 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43306]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:23.925 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:40:23 smithi137 sudo[43306]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:24.178 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43313]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T23:40:24.178 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43313]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:24.178 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43313]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:24.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T23:40:24.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:23 smithi137 ceph-mon[23698]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T23:40:24.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:24.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-20T23:40:24.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:23 smithi144 ceph-mon[26057]: osdmap e22: 3 total, 3 up, 3 in 2024-09-20T23:40:24.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:24.460 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43322]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T23:40:24.460 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43322]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:24.460 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43322]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:24.920 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43351]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T23:40:24.920 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43351]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:24.920 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43351]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:25.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43378]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T23:40:25.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43378]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:25.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:40:24 smithi137 sudo[43378]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:25.440 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43395]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T23:40:25.440 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43395]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:25.440 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43395]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:25.701 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 ceph-mon[26057]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:25.701 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 ceph-mon[26057]: mgrmap e19: smithi137.eurecx(active, since 2m), standbys: smithi144.ygsngz 2024-09-20T23:40:25.701 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 ceph-mon[26057]: osdmap e23: 3 total, 3 up, 3 in 2024-09-20T23:40:25.701 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 ceph-mon[26057]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T23:40:25.701 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28354]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-20T23:40:25.702 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28354]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:25.702 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28354]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:25.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43417]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T23:40:25.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43417]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:25.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 sudo[43417]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:25.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 ceph-mon[23698]: pgmap v76: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:25.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 ceph-mon[23698]: mgrmap e19: smithi137.eurecx(active, since 2m), standbys: smithi144.ygsngz 2024-09-20T23:40:25.786 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 ceph-mon[23698]: osdmap e23: 3 total, 3 up, 3 in 2024-09-20T23:40:25.786 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:25 smithi137 ceph-mon[23698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T23:40:25.961 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28357]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-20T23:40:25.961 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28357]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-09-20T23:40:25.961 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:25 smithi144 sudo[28357]: pam_unix(sudo:session): session closed for user root 2024-09-20T23:40:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:26.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-20T23:40:26.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='client.14298 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:26.775 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:26.776 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:26.776 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:26.776 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:27.641 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:27 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi137 2024-09-20T23:40:27.641 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:27 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi137 to 4288M 2024-09-20T23:40:27.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:27 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi137 2024-09-20T23:40:27.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:27 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi137 to 4288M 2024-09-20T23:40:28.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:28 smithi137 ceph-mon[23698]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:28.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:28 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1991019560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cb33812-30ab-4c68-9ce2-4a51304f6552"}]: dispatch 2024-09-20T23:40:28.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:28 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1991019560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cb33812-30ab-4c68-9ce2-4a51304f6552"}]': finished 2024-09-20T23:40:28.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:28 smithi137 ceph-mon[23698]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T23:40:28.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:28 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:28 smithi144 ceph-mon[26057]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:28 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1991019560' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cb33812-30ab-4c68-9ce2-4a51304f6552"}]: dispatch 2024-09-20T23:40:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:28 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1991019560' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cb33812-30ab-4c68-9ce2-4a51304f6552"}]': finished 2024-09-20T23:40:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:28 smithi144 ceph-mon[26057]: osdmap e24: 4 total, 3 up, 4 in 2024-09-20T23:40:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:28 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:29.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:29 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1808902725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:29 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1808902725' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:30 smithi137 ceph-mon[23698]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:30.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:30 smithi144 ceph-mon[26057]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:32.868 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:32 smithi137 ceph-mon[23698]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-20T23:40:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:32 smithi144 ceph-mon[26057]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-09-20T23:40:34.549 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:34 smithi137 ceph-mon[23698]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-20T23:40:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:34 smithi144 ceph-mon[26057]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-09-20T23:40:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:40:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:35 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:35 smithi137 ceph-mon[23698]: Deploying daemon osd.3 on smithi137 2024-09-20T23:40:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:40:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:35 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:35 smithi144 ceph-mon[26057]: Deploying daemon osd.3 on smithi137 2024-09-20T23:40:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:36 smithi137 ceph-mon[23698]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-20T23:40:36.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:36 smithi144 ceph-mon[26057]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-20T23:40:38.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:38 smithi137 ceph-mon[23698]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-20T23:40:38.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:38 smithi144 ceph-mon[26057]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-09-20T23:40:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:40 smithi137 ceph-mon[23698]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-09-20T23:40:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:40.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:40 smithi144 ceph-mon[26057]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.6 KiB/s rd, 71 KiB/s wr, 5 op/s 2024-09-20T23:40:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:42.835 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:42 smithi137 ceph-mon[23698]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T23:40:42.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:42 smithi144 ceph-mon[26057]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-09-20T23:40:43.483 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 3 on host 'smithi137' 2024-09-20T23:40:43.839 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:43 smithi137 ceph-mon[23698]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:40:43.839 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:43 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:43.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:43 smithi144 ceph-mon[26057]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:40:43.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:43 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:44.173 DEBUG:teuthology.orchestra.run.smithi137:osd.3> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.3.service 2024-09-20T23:40:44.175 INFO:tasks.cephadm:Deploying osd.4 on smithi144 with /dev/vg_nvme/lv_4... 2024-09-20T23:40:44.176 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-20T23:40:44.804 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:44 smithi144 ceph-mon[26057]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:44.805 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:44 smithi144 ceph-mon[26057]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T23:40:44.805 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:44 smithi144 ceph-mon[26057]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T23:40:44.805 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:44 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:44.805 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:44 smithi144 ceph-mon[26057]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:40:44.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-mon[23698]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:44.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-mon[23698]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T23:40:44.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-mon[23698]: osdmap e25: 4 total, 3 up, 4 in 2024-09-20T23:40:44.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:44.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-mon[23698]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:40:44.849 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[45060]: 2024-09-20T23:40:44.545+0000 7f5f953c3700 -1 osd.3 0 waiting for initial osdmap 2024-09-20T23:40:44.850 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:40:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[45060]: 2024-09-20T23:40:44.550+0000 7f5f8dd56700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:40:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:45 smithi137 ceph-mon[23698]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:40:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:45 smithi137 ceph-mon[23698]: osdmap e26: 4 total, 3 up, 4 in 2024-09-20T23:40:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:45 smithi144 ceph-mon[26057]: from='osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-20T23:40:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:45 smithi144 ceph-mon[26057]: osdmap e26: 4 total, 3 up, 4 in 2024-09-20T23:40:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596] boot 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:46.856 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:46.861 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:40:46.868 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:40:46.868 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: osd.3 [v2:172.21.15.137:6826/3785166596,v1:172.21.15.137:6827/3785166596] boot 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: osdmap e27: 4 total, 4 up, 4 in 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:40:46.869 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:46.881 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi144:vg_nvme/lv_4 2024-09-20T23:40:47.638 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:47 smithi137 ceph-mon[23698]: osdmap e28: 4 total, 4 up, 4 in 2024-09-20T23:40:47.846 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:47 smithi144 ceph-mon[26057]: osdmap e28: 4 total, 4 up, 4 in 2024-09-20T23:40:48.648 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:48 smithi144 ceph-mon[26057]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:48 smithi137 ceph-mon[23698]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:49.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:49.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:40:49.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:49.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:49 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='client.24139 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:40:49.814 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:49 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:40:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:50 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi137 2024-09-20T23:40:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:50 smithi137 ceph-mon[23698]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:50 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi137 to 3216M 2024-09-20T23:40:50.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:50 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi137 2024-09-20T23:40:50.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:50 smithi144 ceph-mon[26057]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:50.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:50 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi137 to 3216M 2024-09-20T23:40:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:51 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/2838886141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]: dispatch 2024-09-20T23:40:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:51 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]: dispatch 2024-09-20T23:40:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:51 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]': finished 2024-09-20T23:40:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:51 smithi137 ceph-mon[23698]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T23:40:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:40:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:51 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/2838886141' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]: dispatch 2024-09-20T23:40:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:51 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]: dispatch 2024-09-20T23:40:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:51 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75f786cf-7357-4300-bf40-5cf8361a8440"}]': finished 2024-09-20T23:40:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:51 smithi144 ceph-mon[26057]: osdmap e29: 5 total, 4 up, 5 in 2024-09-20T23:40:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:40:52.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:52 smithi137 ceph-mon[23698]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:52.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:52 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1219220704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:52.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:52 smithi144 ceph-mon[26057]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:52.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:52 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/1219220704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:40:54.813 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:54 smithi144 ceph-mon[26057]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:54.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:54 smithi137 ceph-mon[23698]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:56.682 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:56 smithi144 ceph-mon[26057]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:56.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:56 smithi137 ceph-mon[23698]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:57 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:40:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:57 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:57 smithi137 ceph-mon[23698]: Deploying daemon osd.4 on smithi144 2024-09-20T23:40:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:57 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:40:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:57 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:40:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:57 smithi144 ceph-mon[26057]: Deploying daemon osd.4 on smithi144 2024-09-20T23:40:58.795 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:40:58 smithi144 ceph-mon[26057]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:40:58.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:40:58 smithi137 ceph-mon[23698]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:00.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:00 smithi137 ceph-mon[23698]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:00 smithi144 ceph-mon[26057]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:02 smithi137 ceph-mon[23698]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:03.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:02 smithi144 ceph-mon[26057]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:04 smithi144 ceph-mon[26057]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:05.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:04 smithi137 ceph-mon[23698]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:05.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:05.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:05.775 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 4 on host 'smithi144' 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:06.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:05 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:06.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:41:06.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:41:06.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:06.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:06.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:06.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:05 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:06.385 DEBUG:teuthology.orchestra.run.smithi144:osd.4> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service 2024-09-20T23:41:06.387 INFO:tasks.cephadm:Deploying osd.5 on smithi144 with /dev/vg_nvme/lv_3... 2024-09-20T23:41:06.388 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: from='osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:06 smithi137 ceph-mon[23698]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T23:41:07.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: osdmap e30: 5 total, 4 up, 5 in 2024-09-20T23:41:07.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:07.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: from='osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:07.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-mon[26057]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:07.194 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[30264]: 2024-09-20T23:41:06.874+0000 7fa1bc1f5700 -1 osd.4 0 waiting for initial osdmap 2024-09-20T23:41:07.194 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:41:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[30264]: 2024-09-20T23:41:06.879+0000 7fa1b738d700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:41:08.123 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:07 smithi144 ceph-mon[26057]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:08.123 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:07 smithi144 ceph-mon[26057]: osdmap e31: 5 total, 4 up, 5 in 2024-09-20T23:41:08.123 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:08.123 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:07 smithi137 ceph-mon[23698]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:07 smithi137 ceph-mon[23698]: osdmap e31: 5 total, 4 up, 5 in 2024-09-20T23:41:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612] boot 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:09.079 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:08 smithi144 ceph-mon[26057]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T23:41:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:41:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:41:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-09-20T23:41:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: osd.4 [v2:172.21.15.144:6800/2159317612,v1:172.21.15.144:6801/2159317612] boot 2024-09-20T23:41:09.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: osdmap e32: 5 total, 5 up, 5 in 2024-09-20T23:41:09.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:41:09.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:08 smithi137 ceph-mon[23698]: osdmap e33: 5 total, 5 up, 5 in 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:11.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: osdmap e34: 5 total, 5 up, 5 in 2024-09-20T23:41:11.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:11.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:11.836 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:41:11.855 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi144:vg_nvme/lv_3 2024-09-20T23:41:12.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:11 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi144 2024-09-20T23:41:12.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:11 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi144 to 15936M 2024-09-20T23:41:12.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:11 smithi144 ceph-mon[26057]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T23:41:12.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:41:12.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:41:12.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:11 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi144 2024-09-20T23:41:12.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:11 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi144 to 15936M 2024-09-20T23:41:12.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:11 smithi137 ceph-mon[23698]: osdmap e35: 5 total, 5 up, 5 in 2024-09-20T23:41:12.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:41:12.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:41:13.095 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:12 smithi144 ceph-mon[26057]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:13.095 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:12 smithi144 ceph-mon[26057]: osdmap e36: 5 total, 5 up, 5 in 2024-09-20T23:41:13.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:12 smithi137 ceph-mon[23698]: pgmap v113: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:13.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:12 smithi137 ceph-mon[23698]: osdmap e36: 5 total, 5 up, 5 in 2024-09-20T23:41:14.100 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:41:14.101 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:41:14.101 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:14.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:41:14.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:41:14.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:15.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:14 smithi137 ceph-mon[23698]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:15.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:14 smithi137 ceph-mon[23698]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:15.188 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:14 smithi144 ceph-mon[26057]: pgmap v115: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:15.188 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:14 smithi144 ceph-mon[26057]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:17.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:41:17.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/3785032579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]: dispatch 2024-09-20T23:41:17.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]: dispatch 2024-09-20T23:41:17.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]': finished 2024-09-20T23:41:17.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T23:41:17.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:17.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:16 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/3515114082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:41:17.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/3785032579' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]: dispatch 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]: dispatch 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d7d6be5-d25b-4b70-814f-49b571dc416d"}]': finished 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: osdmap e37: 6 total, 5 up, 6 in 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:17.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:16 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/3515114082' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:41:19.095 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:18 smithi144 ceph-mon[26057]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-20T23:41:19.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:18 smithi137 ceph-mon[23698]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-09-20T23:41:21.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:20 smithi137 ceph-mon[23698]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:41:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:20 smithi144 ceph-mon[26057]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:41:23.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:22 smithi137 ceph-mon[23698]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-20T23:41:23.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:41:23.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:23.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:22 smithi137 ceph-mon[23698]: Deploying daemon osd.5 on smithi144 2024-09-20T23:41:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:22 smithi144 ceph-mon[26057]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 54 KiB/s, 0 objects/s recovering 2024-09-20T23:41:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:41:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:22 smithi144 ceph-mon[26057]: Deploying daemon osd.5 on smithi144 2024-09-20T23:41:25.117 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:24 smithi144 ceph-mon[26057]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:41:25.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:24 smithi137 ceph-mon[23698]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:41:27.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:26 smithi137 ceph-mon[23698]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:27.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:27.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:27.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:27.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:26 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:26 smithi144 ceph-mon[26057]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:26 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:29.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:28 smithi137 ceph-mon[23698]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:29.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:28 smithi144 ceph-mon[26057]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:29.573 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 5 on host 'smithi144' 2024-09-20T23:41:30.511 DEBUG:teuthology.orchestra.run.smithi144:osd.5> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.5.service 2024-09-20T23:41:30.514 INFO:tasks.cephadm:Deploying osd.6 on smithi144 with /dev/vg_nvme/lv_2... 2024-09-20T23:41:30.514 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-20T23:41:30.832 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:30 smithi144 ceph-mon[26057]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:30.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:30 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:30.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:30 smithi144 ceph-mon[26057]: from='osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:41:30.833 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:30 smithi144 ceph-mon[26057]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:41:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:30 smithi137 ceph-mon[23698]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:30 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:30 smithi137 ceph-mon[23698]: from='osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:41:30.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:30 smithi137 ceph-mon[23698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:41:32.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-mon[26057]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T23:41:32.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-mon[26057]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T23:41:32.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:32.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-mon[26057]: from='osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:32.121 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-mon[26057]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:32.121 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[33207]: 2024-09-20T23:41:31.944+0000 7f80bf50a700 -1 osd.5 0 waiting for initial osdmap 2024-09-20T23:41:32.121 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:41:31 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[33207]: 2024-09-20T23:41:31.950+0000 7f80b96a0700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:41:32.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:31 smithi137 ceph-mon[23698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T23:41:32.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:31 smithi137 ceph-mon[23698]: osdmap e38: 6 total, 5 up, 6 in 2024-09-20T23:41:32.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:31 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:32.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:31 smithi137 ceph-mon[23698]: from='osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:32.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:31 smithi137 ceph-mon[23698]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:33.149 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396] boot 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.150 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:32 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:33.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-09-20T23:41:33.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: osdmap e39: 6 total, 5 up, 6 in 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: osd.5 [v2:172.21.15.144:6808/3485410396,v1:172.21.15.144:6809/3485410396] boot 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: osdmap e40: 6 total, 6 up, 6 in 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:41:33.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:32 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:34.084 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:33 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:41:34.084 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:33 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:41:34.084 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:33 smithi144 ceph-mon[26057]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T23:41:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:33 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:41:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:33 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:41:34.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:33 smithi137 ceph-mon[23698]: osdmap e41: 6 total, 6 up, 6 in 2024-09-20T23:41:34.734 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:41:34.759 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi144:vg_nvme/lv_2 2024-09-20T23:41:35.049 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:34 smithi144 ceph-mon[26057]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:35.049 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:34 smithi144 ceph-mon[26057]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T23:41:35.049 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:34 smithi144 ceph-mon[26057]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T23:41:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:34 smithi137 ceph-mon[23698]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:34 smithi137 ceph-mon[23698]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T23:41:35.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:34 smithi137 ceph-mon[23698]: osdmap e42: 6 total, 6 up, 6 in 2024-09-20T23:41:37.277 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:36 smithi144 ceph-mon[26057]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:37.278 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:36 smithi144 ceph-mon[26057]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T23:41:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:36 smithi137 ceph-mon[23698]: pgmap v132: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:36 smithi137 ceph-mon[23698]: osdmap e43: 6 total, 6 up, 6 in 2024-09-20T23:41:38.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T23:41:38.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi144 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi144 to 7968M 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:41:38.258 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: osdmap e44: 6 total, 6 up, 6 in 2024-09-20T23:41:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi144 2024-09-20T23:41:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi144 to 7968M 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:41:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:39.255 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:39 smithi144 ceph-mon[26057]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:39.255 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:39 smithi144 ceph-mon[26057]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:39 smithi137 ceph-mon[23698]: pgmap v135: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:39 smithi137 ceph-mon[23698]: from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:41:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:40 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/86935901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]: dispatch 2024-09-20T23:41:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:40 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]: dispatch 2024-09-20T23:41:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:40 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]': finished 2024-09-20T23:41:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:40 smithi137 ceph-mon[23698]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T23:41:40.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:40 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/86935901' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]: dispatch 2024-09-20T23:41:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:40 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]: dispatch 2024-09-20T23:41:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:40 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1e6c3193-3024-40cd-a997-c043574eb65d"}]': finished 2024-09-20T23:41:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:40 smithi144 ceph-mon[26057]: osdmap e45: 7 total, 6 up, 7 in 2024-09-20T23:41:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:41 smithi137 ceph-mon[23698]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-20T23:41:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:41 smithi137 ceph-mon[23698]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T23:41:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:41 smithi137 ceph-mon[23698]: Cluster is now healthy 2024-09-20T23:41:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:41 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/1155856454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:41:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:41 smithi144 ceph-mon[26057]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 88 KiB/s, 0 objects/s recovering 2024-09-20T23:41:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:41 smithi144 ceph-mon[26057]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T23:41:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:41 smithi144 ceph-mon[26057]: Cluster is now healthy 2024-09-20T23:41:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:41 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/1155856454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:41:43.261 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:43 smithi144 ceph-mon[26057]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:41:43.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:43 smithi137 ceph-mon[23698]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:41:45.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:45 smithi144 ceph-mon[26057]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-20T23:41:45.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:45 smithi137 ceph-mon[23698]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-09-20T23:41:46.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:41:46.385 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:46.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:41:46.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:47.303 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:47 smithi144 ceph-mon[26057]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-20T23:41:47.303 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:47 smithi144 ceph-mon[26057]: Deploying daemon osd.6 on smithi144 2024-09-20T23:41:47.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:47 smithi137 ceph-mon[23698]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-09-20T23:41:47.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:47 smithi137 ceph-mon[23698]: Deploying daemon osd.6 on smithi144 2024-09-20T23:41:49.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:49 smithi137 ceph-mon[23698]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:41:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:49 smithi144 ceph-mon[26057]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:41:51.131 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:51 smithi137 ceph-mon[23698]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:51.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:51.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:51.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:51.132 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:51.322 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:51 smithi144 ceph-mon[26057]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:51.323 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:41:51.323 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:51.323 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:41:51.323 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:41:53.178 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:53 smithi144 ceph-mon[26057]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:53.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:53 smithi137 ceph-mon[23698]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:54.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:54 smithi144 ceph-mon[26057]: from='osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:41:54.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:54 smithi144 ceph-mon[26057]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:41:54.333 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 6 on host 'smithi144' 2024-09-20T23:41:54.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:54 smithi137 ceph-mon[23698]: from='osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:41:54.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:54 smithi137 ceph-mon[23698]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:41:55.256 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:55.256 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T23:41:55.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T23:41:55.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:55.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: from='osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:55.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:55.257 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:55.277 DEBUG:teuthology.orchestra.run.smithi144:osd.6> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.6.service 2024-09-20T23:41:55.279 INFO:tasks.cephadm:Deploying osd.7 on smithi144 with /dev/vg_nvme/lv_1... 2024-09-20T23:41:55.279 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: osdmap e46: 7 total, 6 up, 7 in 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: from='osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:41:55.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:56.280 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:56 smithi144 ceph-mon[26057]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:56.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:56 smithi144 ceph-mon[26057]: osdmap e47: 7 total, 6 up, 7 in 2024-09-20T23:41:56.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:56.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:56.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:56 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:56 smithi137 ceph-mon[23698]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:41:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:56 smithi137 ceph-mon[23698]: osdmap e47: 7 total, 6 up, 7 in 2024-09-20T23:41:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:56 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:41:57.379 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:41:57.379 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:41:57.379 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:57.379 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455] boot 2024-09-20T23:41:57.380 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T23:41:57.380 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:57 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: osd.6 [v2:172.21.15.144:6816/1717611455,v1:172.21.15.144:6817/1717611455] boot 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: osdmap e48: 7 total, 7 up, 7 in 2024-09-20T23:41:57.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:57 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:41:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:58 smithi144 ceph-mon[26057]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T23:41:58.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:58 smithi137 ceph-mon[23698]: osdmap e49: 7 total, 7 up, 7 in 2024-09-20T23:41:58.808 INFO:teuthology.orchestra.run.smithi144.stdout: 2024-09-20T23:41:58.832 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch daemon add osd smithi144:vg_nvme/lv_1 2024-09-20T23:41:59.382 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:59 smithi144 ceph-mon[26057]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:41:59.382 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:41:59 smithi144 ceph-mon[26057]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T23:41:59.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:59 smithi137 ceph-mon[23698]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:41:59.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:41:59 smithi137 ceph-mon[23698]: osdmap e50: 7 total, 7 up, 7 in 2024-09-20T23:42:00.229 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:00 smithi144 ceph-mon[26057]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T23:42:00.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:00 smithi137 ceph-mon[23698]: osdmap e51: 7 total, 7 up, 7 in 2024-09-20T23:42:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:01 smithi144 ceph-mon[26057]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:01 smithi144 ceph-mon[26057]: osdmap e52: 7 total, 7 up, 7 in 2024-09-20T23:42:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:42:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:42:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:01 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:01 smithi137 ceph-mon[23698]: pgmap v153: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:01 smithi137 ceph-mon[23698]: osdmap e52: 7 total, 7 up, 7 in 2024-09-20T23:42:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-20T23:42:01.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-20T23:42:01.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:01 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:02.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:02.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:02.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:02 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='client.24193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi144:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:02.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:02 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:03.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:03 smithi137 ceph-mon[23698]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:03.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:03 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi144 2024-09-20T23:42:03.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:03 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi144 to 5312M 2024-09-20T23:42:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:03 smithi144 ceph-mon[26057]: pgmap v155: 1 pgs: 1 remapped+peering; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:03 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi144 2024-09-20T23:42:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:03 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi144 to 5312M 2024-09-20T23:42:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:04 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/2392646335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]: dispatch 2024-09-20T23:42:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:04 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]: dispatch 2024-09-20T23:42:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:04 smithi137 ceph-mon[23698]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]': finished 2024-09-20T23:42:04.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:04 smithi137 ceph-mon[23698]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T23:42:04.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:04.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:04 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/2392646335' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]: dispatch 2024-09-20T23:42:04.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:04 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]: dispatch 2024-09-20T23:42:04.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:04 smithi144 ceph-mon[26057]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e0a024f5-271a-40ef-a5fe-843a33b68df7"}]': finished 2024-09-20T23:42:04.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:04 smithi144 ceph-mon[26057]: osdmap e53: 8 total, 7 up, 8 in 2024-09-20T23:42:04.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:05.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:05 smithi137 ceph-mon[23698]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:42:05.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:05 smithi137 ceph-mon[23698]: from='client.? 172.21.15.144:0/4138936503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:42:05.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:05 smithi144 ceph-mon[26057]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-09-20T23:42:05.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:05 smithi144 ceph-mon[26057]: from='client.? 172.21.15.144:0/4138936503' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-20T23:42:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:07 smithi144 ceph-mon[26057]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-20T23:42:07.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:07 smithi137 ceph-mon[23698]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 82 KiB/s, 0 objects/s recovering 2024-09-20T23:42:09.426 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:09 smithi144 ceph-mon[26057]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:42:09.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:09 smithi137 ceph-mon[23698]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:42:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:42:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:10.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:42:10.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:11.346 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:11 smithi144 ceph-mon[26057]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-20T23:42:11.346 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:11 smithi144 ceph-mon[26057]: Deploying daemon osd.7 on smithi144 2024-09-20T23:42:11.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:11 smithi137 ceph-mon[23698]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-20T23:42:11.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:11 smithi137 ceph-mon[23698]: Deploying daemon osd.7 on smithi144 2024-09-20T23:42:12.353 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:42:12.353 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:42:12.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:42:12.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:42:13.353 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:13 smithi144 ceph-mon[26057]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:42:13.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:13 smithi137 ceph-mon[23698]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-09-20T23:42:15.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:15 smithi137 ceph-mon[23698]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:15.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:15 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:15.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:15 smithi144 ceph-mon[26057]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:15.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:15 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:16.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:16.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:16.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:16 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:16 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:16 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:16 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:17.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:17 smithi144 ceph-mon[26057]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:17.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:17 smithi137 ceph-mon[23698]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:18.950 INFO:teuthology.orchestra.run.smithi144.stdout:Created osd(s) 7 on host 'smithi144' 2024-09-20T23:42:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:19 smithi144 ceph-mon[26057]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:19 smithi144 ceph-mon[26057]: from='osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:42:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:19 smithi144 ceph-mon[26057]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:42:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:19 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:19.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:19 smithi137 ceph-mon[23698]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:19.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:19 smithi137 ceph-mon[23698]: from='osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:42:19.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:19 smithi137 ceph-mon[23698]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:42:19.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:19 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:19.705 DEBUG:teuthology.orchestra.run.smithi144:osd.7> sudo journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.7.service 2024-09-20T23:42:19.707 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-20T23:42:19.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd stat -f json 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: from='osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:42:20.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:20 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:20.436 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T23:42:20.436 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: osdmap e54: 8 total, 7 up, 8 in 2024-09-20T23:42:20.436 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:20.437 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: from='osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:42:20.437 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:42:20.437 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:20.437 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[39206]: 2024-09-20T23:42:20.149+0000 7f987cba9700 -1 osd.7 0 waiting for initial osdmap 2024-09-20T23:42:20.437 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:42:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[39206]: 2024-09-20T23:42:20.156+0000 7f9877540700 -1 osd.7 55 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:42:21.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:21 smithi137 ceph-mon[23698]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:21.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:21 smithi137 ceph-mon[23698]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:42:21.271 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:21 smithi137 ceph-mon[23698]: osdmap e55: 8 total, 7 up, 8 in 2024-09-20T23:42:21.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:21.272 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:21 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:21.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:21 smithi144 ceph-mon[26057]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-09-20T23:42:21.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:21 smithi144 ceph-mon[26057]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]': finished 2024-09-20T23:42:21.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:21 smithi144 ceph-mon[26057]: osdmap e55: 8 total, 7 up, 8 in 2024-09-20T23:42:21.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:21.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:21 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:21.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: purged_snaps scrub starts 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: purged_snaps scrub ok 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010] boot 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:22.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/673779495' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T23:42:22.311 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:22 smithi137 ceph-mon[23698]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T23:42:22.333 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":56,"num_osds":8,"num_up_osds":8,"osd_up_since":1726875741,"num_in_osds":8,"osd_in_since":1726875723,"num_remapped_pgs":1} 2024-09-20T23:42:22.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: purged_snaps scrub starts 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: purged_snaps scrub ok 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: osd.7 [v2:172.21.15.144:6824/4206686010,v1:172.21.15.144:6825/4206686010] boot 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: osdmap e56: 8 total, 8 up, 8 in 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/673779495' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-20T23:42:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:22 smithi144 ceph-mon[26057]: osdmap e57: 8 total, 8 up, 8 in 2024-09-20T23:42:22.551 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:23.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:23.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:23 smithi137 ceph-mon[23698]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T23:42:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:42:23.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:42:23.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:23 smithi144 ceph-mon[26057]: osdmap e58: 8 total, 8 up, 8 in 2024-09-20T23:42:24.373 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:42:24.373 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":59,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","created":"2024-09-20T23:36:15.044935+0000","modified":"2024-09-20T23:42:24.157516+0000","last_up_change":"2024-09-20T23:42:21.142672+0000","last_in_change":"2024-09-20T23:42:03.735419+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T23:40:21.666783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cfd14532-0384-4863-ae9b-eaf2bc30783f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6803","nonce":3729731545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6805","nonce":3729731545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6809","nonce":3729731545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6807","nonce":3729731545}]},"public_addr":"172.21.15.137:6803/3729731545","cluster_addr":"172.21.15.137:6805/3729731545","heartbeat_back_addr":"172.21.15.137:6809/3729731545","heartbeat_front_addr":"172.21.15.137:6807/3729731545","state":["exists","up"]},{"osd":1,"uuid":"cad3fdbc-67f4-4f11-9532-64cf3fb7910d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6811","nonce":1601068046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6813","nonce":1601068046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6817","nonce":1601068046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6815","nonce":1601068046}]},"public_addr":"172.21.15.137:6811/1601068046","cluster_addr":"172.21.15.137:6813/1601068046","heartbeat_back_addr":"172.21.15.137:6817/1601068046","heartbeat_front_addr":"172.21.15.137:6815/1601068046","state":["exists","up"]},{"osd":2,"uuid":"bc60058a-b022-4ab7-b855-6341004902cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6819","nonce":3517536294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6821","nonce":3517536294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6825","nonce":3517536294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6823","nonce":3517536294}]},"public_addr":"172.21.15.137:6819/3517536294","cluster_addr":"172.21.15.137:6821/3517536294","heartbeat_back_addr":"172.21.15.137:6825/3517536294","heartbeat_front_addr":"172.21.15.137:6823/3517536294","state":["exists","up"]},{"osd":3,"uuid":"7cb33812-30ab-4c68-9ce2-4a51304f6552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6827","nonce":3785166596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6829","nonce":3785166596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6833","nonce":3785166596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6831","nonce":3785166596}]},"public_addr":"172.21.15.137:6827/3785166596","cluster_addr":"172.21.15.137:6829/3785166596","heartbeat_back_addr":"172.21.15.137:6833/3785166596","heartbeat_front_addr":"172.21.15.137:6831/3785166596","state":["exists","up"]},{"osd":4,"uuid":"75f786cf-7357-4300-bf40-5cf8361a8440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6801","nonce":2159317612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6803","nonce":2159317612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6807","nonce":2159317612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6805","nonce":2159317612}]},"public_addr":"172.21.15.144:6801/2159317612","cluster_addr":"172.21.15.144:6803/2159317612","heartbeat_back_addr":"172.21.15.144:6807/2159317612","heartbeat_front_addr":"172.21.15.144:6805/2159317612","state":["exists","up"]},{"osd":5,"uuid":"7d7d6be5-d25b-4b70-814f-49b571dc416d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6809","nonce":3485410396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6811","nonce":3485410396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6815","nonce":3485410396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6813","nonce":3485410396}]},"public_addr":"172.21.15.144:6809/3485410396","cluster_addr":"172.21.15.144:6811/3485410396","heartbeat_back_addr":"172.21.15.144:6815/3485410396","heartbeat_front_addr":"172.21.15.144:6813/3485410396","state":["exists","up"]},{"osd":6,"uuid":"1e6c3193-3024-40cd-a997-c043574eb65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6817","nonce":1717611455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6819","nonce":1717611455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6823","nonce":1717611455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6821","nonce":1717611455}]},"public_addr":"172.21.15.144:6817/1717611455","cluster_addr":"172.21.15.144:6819/1717611455","heartbeat_back_addr":"172.21.15.144:6823/1717611455","heartbeat_front_addr":"172.21.15.144:6821/1717611455","state":["exists","up"]},{"osd":7,"uuid":"e0a024f5-271a-40ef-a5fe-843a33b68df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6825","nonce":4206686010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6827","nonce":4206686010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6831","nonce":4206686010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6829","nonce":4206686010}]},"public_addr":"172.21.15.144:6825/4206686010","cluster_addr":"172.21.15.144:6827/4206686010","heartbeat_back_addr":"172.21.15.144:6831/4206686010","heartbeat_front_addr":"172.21.15.144:6829/4206686010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:20.569249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:51.823732+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:17.932449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:43.602583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:05.990757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:31.047590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:54.916281+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:42:19.211389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/2546958348":"2024-09-21T23:38:11.297896+0000","172.21.15.137:6801/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/1488967328":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3580148197":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3016470345":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6801/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3909251190":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/358720658":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3998844218":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3383681779":"2024-09-21T23:36:46.535745+0000","172.21.15.137:6801/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3681805263":"2024-09-21T23:36:46.535745+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T23:42:24.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:24 smithi137 ceph-mon[23698]: Detected new or changed devices on smithi144 2024-09-20T23:42:24.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:24 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi144 to 3984M 2024-09-20T23:42:24.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:24 smithi144 ceph-mon[26057]: Detected new or changed devices on smithi144 2024-09-20T23:42:24.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:24 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi144 to 3984M 2024-09-20T23:42:24.989 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-20T23:40:21.666783+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-09-20T23:42:24.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-20T23:42:25.200 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:25.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:25 smithi137 ceph-mon[23698]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:25.322 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:25 smithi137 ceph-mon[23698]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T23:42:25.322 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:25 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/889528795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:25 smithi144 ceph-mon[26057]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:25 smithi144 ceph-mon[26057]: osdmap e59: 8 total, 8 up, 8 in 2024-09-20T23:42:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:25 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/889528795' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:26.313 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:26 smithi137 ceph-mon[23698]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T23:42:26.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:26 smithi144 ceph-mon[26057]: osdmap e60: 8 total, 8 up, 8 in 2024-09-20T23:42:27.029 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 1 2024-09-20T23:42:27.335 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:27 smithi137 ceph-mon[23698]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:27.335 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:27 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2832289125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T23:42:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:27 smithi144 ceph-mon[26057]: pgmap v175: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:27 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2832289125' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-20T23:42:27.779 INFO:tasks.cephadm:Setting up client nodes... 2024-09-20T23:42:27.780 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-20T23:42:29.372 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:29 smithi137 ceph-mon[23698]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-20T23:42:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:29 smithi144 ceph-mon[26057]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 98 KiB/s, 0 objects/s recovering 2024-09-20T23:42:29.834 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-09-20T23:42:29.834 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQBlCO5mkmRxMRAAiE2tAnAHUNRi3Gg/USCBNg== 2024-09-20T23:42:30.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:30 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1956091258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T23:42:30.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:30 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1956091258' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T23:42:30.533 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:30 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1956091258' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-20T23:42:30.533 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:30 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1956091258' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-20T23:42:30.559 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-20T23:42:30.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-20T23:42:30.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-20T23:42:30.599 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-20T23:42:30.599 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-20T23:42:30.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph mgr dump --format=json 2024-09-20T23:42:30.846 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:31.262 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:31 smithi137 ceph-mon[23698]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T23:42:31.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:31 smithi144 ceph-mon[26057]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-09-20T23:42:32.573 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:42:33.328 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":19,"active_gid":14206,"active_name":"smithi137.eurecx","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6800","nonce":328477604},{"type":"v1","addr":"172.21.15.137:6801","nonce":328477604}]},"active_addr":"172.21.15.137:6801/328477604","active_change":"2024-09-20T23:38:11.298205+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14224,"name":"smithi144.ygsngz","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.137:8443/","prometheus":"http://172.21.15.137:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":2129186664}]},{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3670583884}]},{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":388888943}]},{"addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1470600426}]}]}} 2024-09-20T23:42:33.331 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-20T23:42:33.331 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-20T23:42:33.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T23:42:33.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:33 smithi144 ceph-mon[26057]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:42:33.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:33 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1533016599' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T23:42:33.539 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:33.609 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:33 smithi137 ceph-mon[23698]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-09-20T23:42:33.609 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:33 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1533016599' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-20T23:42:35.336 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:42:35.337 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":60,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","created":"2024-09-20T23:36:15.044935+0000","modified":"2024-09-20T23:42:25.163941+0000","last_up_change":"2024-09-20T23:42:21.142672+0000","last_in_change":"2024-09-20T23:42:03.735419+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T23:40:21.666783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cfd14532-0384-4863-ae9b-eaf2bc30783f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6803","nonce":3729731545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6805","nonce":3729731545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6809","nonce":3729731545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6807","nonce":3729731545}]},"public_addr":"172.21.15.137:6803/3729731545","cluster_addr":"172.21.15.137:6805/3729731545","heartbeat_back_addr":"172.21.15.137:6809/3729731545","heartbeat_front_addr":"172.21.15.137:6807/3729731545","state":["exists","up"]},{"osd":1,"uuid":"cad3fdbc-67f4-4f11-9532-64cf3fb7910d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6811","nonce":1601068046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6813","nonce":1601068046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6817","nonce":1601068046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6815","nonce":1601068046}]},"public_addr":"172.21.15.137:6811/1601068046","cluster_addr":"172.21.15.137:6813/1601068046","heartbeat_back_addr":"172.21.15.137:6817/1601068046","heartbeat_front_addr":"172.21.15.137:6815/1601068046","state":["exists","up"]},{"osd":2,"uuid":"bc60058a-b022-4ab7-b855-6341004902cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6819","nonce":3517536294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6821","nonce":3517536294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6825","nonce":3517536294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6823","nonce":3517536294}]},"public_addr":"172.21.15.137:6819/3517536294","cluster_addr":"172.21.15.137:6821/3517536294","heartbeat_back_addr":"172.21.15.137:6825/3517536294","heartbeat_front_addr":"172.21.15.137:6823/3517536294","state":["exists","up"]},{"osd":3,"uuid":"7cb33812-30ab-4c68-9ce2-4a51304f6552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6827","nonce":3785166596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6829","nonce":3785166596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6833","nonce":3785166596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6831","nonce":3785166596}]},"public_addr":"172.21.15.137:6827/3785166596","cluster_addr":"172.21.15.137:6829/3785166596","heartbeat_back_addr":"172.21.15.137:6833/3785166596","heartbeat_front_addr":"172.21.15.137:6831/3785166596","state":["exists","up"]},{"osd":4,"uuid":"75f786cf-7357-4300-bf40-5cf8361a8440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6801","nonce":2159317612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6803","nonce":2159317612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6807","nonce":2159317612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6805","nonce":2159317612}]},"public_addr":"172.21.15.144:6801/2159317612","cluster_addr":"172.21.15.144:6803/2159317612","heartbeat_back_addr":"172.21.15.144:6807/2159317612","heartbeat_front_addr":"172.21.15.144:6805/2159317612","state":["exists","up"]},{"osd":5,"uuid":"7d7d6be5-d25b-4b70-814f-49b571dc416d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6809","nonce":3485410396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6811","nonce":3485410396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6815","nonce":3485410396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6813","nonce":3485410396}]},"public_addr":"172.21.15.144:6809/3485410396","cluster_addr":"172.21.15.144:6811/3485410396","heartbeat_back_addr":"172.21.15.144:6815/3485410396","heartbeat_front_addr":"172.21.15.144:6813/3485410396","state":["exists","up"]},{"osd":6,"uuid":"1e6c3193-3024-40cd-a997-c043574eb65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6817","nonce":1717611455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6819","nonce":1717611455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6823","nonce":1717611455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6821","nonce":1717611455}]},"public_addr":"172.21.15.144:6817/1717611455","cluster_addr":"172.21.15.144:6819/1717611455","heartbeat_back_addr":"172.21.15.144:6823/1717611455","heartbeat_front_addr":"172.21.15.144:6821/1717611455","state":["exists","up"]},{"osd":7,"uuid":"e0a024f5-271a-40ef-a5fe-843a33b68df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6825","nonce":4206686010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6827","nonce":4206686010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6831","nonce":4206686010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6829","nonce":4206686010}]},"public_addr":"172.21.15.144:6825/4206686010","cluster_addr":"172.21.15.144:6827/4206686010","heartbeat_back_addr":"172.21.15.144:6831/4206686010","heartbeat_front_addr":"172.21.15.144:6829/4206686010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:20.569249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:51.823732+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:17.932449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:43.602583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:05.990757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:31.047590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:54.916281+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:42:19.211389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/2546958348":"2024-09-21T23:38:11.297896+0000","172.21.15.137:6801/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/1488967328":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3580148197":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3016470345":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6801/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3909251190":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/358720658":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3998844218":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3383681779":"2024-09-21T23:36:46.535745+0000","172.21.15.137:6801/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3681805263":"2024-09-21T23:36:46.535745+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T23:42:35.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:35 smithi144 ceph-mon[26057]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-20T23:42:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:35 smithi137 ceph-mon[23698]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-09-20T23:42:36.045 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-20T23:42:36.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd dump --format=json 2024-09-20T23:42:36.255 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:36.377 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:36 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2084750519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:36 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2084750519' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:37 smithi137 ceph-mon[23698]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-20T23:42:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:37 smithi144 ceph-mon[26057]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 50 KiB/s, 0 objects/s recovering 2024-09-20T23:42:38.123 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:42:38.123 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":60,"fsid":"1150c2f6-77a9-11ef-baf5-efdc52797490","created":"2024-09-20T23:36:15.044935+0000","modified":"2024-09-20T23:42:25.163941+0000","last_up_change":"2024-09-20T23:42:21.142672+0000","last_in_change":"2024-09-20T23:42:03.735419+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-20T23:40:21.666783+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cfd14532-0384-4863-ae9b-eaf2bc30783f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6802","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6803","nonce":3729731545}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6804","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6805","nonce":3729731545}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6808","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6809","nonce":3729731545}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6806","nonce":3729731545},{"type":"v1","addr":"172.21.15.137:6807","nonce":3729731545}]},"public_addr":"172.21.15.137:6803/3729731545","cluster_addr":"172.21.15.137:6805/3729731545","heartbeat_back_addr":"172.21.15.137:6809/3729731545","heartbeat_front_addr":"172.21.15.137:6807/3729731545","state":["exists","up"]},{"osd":1,"uuid":"cad3fdbc-67f4-4f11-9532-64cf3fb7910d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":35,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6810","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6811","nonce":1601068046}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6812","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6813","nonce":1601068046}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6816","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6817","nonce":1601068046}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6814","nonce":1601068046},{"type":"v1","addr":"172.21.15.137:6815","nonce":1601068046}]},"public_addr":"172.21.15.137:6811/1601068046","cluster_addr":"172.21.15.137:6813/1601068046","heartbeat_back_addr":"172.21.15.137:6817/1601068046","heartbeat_front_addr":"172.21.15.137:6815/1601068046","state":["exists","up"]},{"osd":2,"uuid":"bc60058a-b022-4ab7-b855-6341004902cd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6818","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6819","nonce":3517536294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6820","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6821","nonce":3517536294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6824","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6825","nonce":3517536294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6822","nonce":3517536294},{"type":"v1","addr":"172.21.15.137:6823","nonce":3517536294}]},"public_addr":"172.21.15.137:6819/3517536294","cluster_addr":"172.21.15.137:6821/3517536294","heartbeat_back_addr":"172.21.15.137:6825/3517536294","heartbeat_front_addr":"172.21.15.137:6823/3517536294","state":["exists","up"]},{"osd":3,"uuid":"7cb33812-30ab-4c68-9ce2-4a51304f6552","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6826","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6827","nonce":3785166596}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6828","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6829","nonce":3785166596}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6832","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6833","nonce":3785166596}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.137:6830","nonce":3785166596},{"type":"v1","addr":"172.21.15.137:6831","nonce":3785166596}]},"public_addr":"172.21.15.137:6827/3785166596","cluster_addr":"172.21.15.137:6829/3785166596","heartbeat_back_addr":"172.21.15.137:6833/3785166596","heartbeat_front_addr":"172.21.15.137:6831/3785166596","state":["exists","up"]},{"osd":4,"uuid":"75f786cf-7357-4300-bf40-5cf8361a8440","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6800","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6801","nonce":2159317612}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6802","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6803","nonce":2159317612}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6806","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6807","nonce":2159317612}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6804","nonce":2159317612},{"type":"v1","addr":"172.21.15.144:6805","nonce":2159317612}]},"public_addr":"172.21.15.144:6801/2159317612","cluster_addr":"172.21.15.144:6803/2159317612","heartbeat_back_addr":"172.21.15.144:6807/2159317612","heartbeat_front_addr":"172.21.15.144:6805/2159317612","state":["exists","up"]},{"osd":5,"uuid":"7d7d6be5-d25b-4b70-814f-49b571dc416d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6808","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6809","nonce":3485410396}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6810","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6811","nonce":3485410396}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6814","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6815","nonce":3485410396}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6812","nonce":3485410396},{"type":"v1","addr":"172.21.15.144:6813","nonce":3485410396}]},"public_addr":"172.21.15.144:6809/3485410396","cluster_addr":"172.21.15.144:6811/3485410396","heartbeat_back_addr":"172.21.15.144:6815/3485410396","heartbeat_front_addr":"172.21.15.144:6813/3485410396","state":["exists","up"]},{"osd":6,"uuid":"1e6c3193-3024-40cd-a997-c043574eb65d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":51,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6816","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6817","nonce":1717611455}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6818","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6819","nonce":1717611455}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6822","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6823","nonce":1717611455}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6820","nonce":1717611455},{"type":"v1","addr":"172.21.15.144:6821","nonce":1717611455}]},"public_addr":"172.21.15.144:6817/1717611455","cluster_addr":"172.21.15.144:6819/1717611455","heartbeat_back_addr":"172.21.15.144:6823/1717611455","heartbeat_front_addr":"172.21.15.144:6821/1717611455","state":["exists","up"]},{"osd":7,"uuid":"e0a024f5-271a-40ef-a5fe-843a33b68df7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":56,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6824","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6825","nonce":4206686010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6826","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6827","nonce":4206686010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6830","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6831","nonce":4206686010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.144:6828","nonce":4206686010},{"type":"v1","addr":"172.21.15.144:6829","nonce":4206686010}]},"public_addr":"172.21.15.144:6825/4206686010","cluster_addr":"172.21.15.144:6827/4206686010","heartbeat_back_addr":"172.21.15.144:6831/4206686010","heartbeat_front_addr":"172.21.15.144:6829/4206686010","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:20.569249+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:39:51.823732+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:17.932449+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:40:43.602583+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:05.990757+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:31.047590+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:41:54.916281+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-09-20T23:42:19.211389+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/2546958348":"2024-09-21T23:38:11.297896+0000","172.21.15.137:6801/3958234312":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/1488967328":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3580148197":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3016470345":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6801/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:0/3909251190":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/776350976":"2024-09-21T23:37:15.039586+0000","172.21.15.137:6800/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/358720658":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3998844218":"2024-09-21T23:38:11.297896+0000","172.21.15.137:0/3383681779":"2024-09-21T23:36:46.535745+0000","172.21.15.137:6801/1549548829":"2024-09-21T23:36:46.535745+0000","172.21.15.137:0/3681805263":"2024-09-21T23:36:46.535745+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-20T23:42:38.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:38 smithi137 ceph-mon[23698]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-09-20T23:42:38.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:38 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2317258881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:38.840 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-20T23:42:38.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-20T23:42:38.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-20T23:42:38.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-20T23:42:38.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-20T23:42:38.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-20T23:42:38.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-20T23:42:38.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-20T23:42:38.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:38 smithi144 ceph-mon[26057]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-09-20T23:42:38.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:38 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2317258881' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-20T23:42:39.239 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.242 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.243 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.243 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.243 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.247 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.256 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:39.263 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:40.743 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:40 smithi137 ceph-mon[23698]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:40 smithi144 ceph-mon[26057]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:42 smithi137 ceph-mon[23698]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:42.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:42 smithi144 ceph-mon[26057]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:44.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:44 smithi137 ceph-mon[23698]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:44.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:44 smithi144 ceph-mon[26057]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:46.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:46 smithi137 ceph-mon[23698]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:46 smithi144 ceph-mon[26057]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:48.844 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:48 smithi137 ceph-mon[23698]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:48 smithi144 ceph-mon[26057]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:50.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:50 smithi137 ceph-mon[23698]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:50.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:50 smithi144 ceph-mon[26057]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:51.949 INFO:teuthology.orchestra.run.smithi137.stdout:60129542183 2024-09-20T23:42:51.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-20T23:42:52.702 INFO:teuthology.orchestra.run.smithi137.stdout:171798691858 2024-09-20T23:42:52.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-20T23:42:52.780 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:52 smithi137 ceph-mon[23698]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:52.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:52 smithi144 ceph-mon[26057]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:54.183 INFO:teuthology.orchestra.run.smithi137.stdout:206158430222 2024-09-20T23:42:54.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-20T23:42:54.678 INFO:teuthology.orchestra.run.smithi137.stdout:115964117020 2024-09-20T23:42:54.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-20T23:42:54.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:54 smithi137 ceph-mon[23698]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:54 smithi144 ceph-mon[26057]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:55.124 INFO:teuthology.orchestra.run.smithi137.stdout:81604378658 2024-09-20T23:42:55.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-20T23:42:55.625 INFO:teuthology.orchestra.run.smithi137.stdout:38654705709 2024-09-20T23:42:55.625 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-20T23:42:56.052 INFO:teuthology.orchestra.run.smithi137.stdout:137438953496 2024-09-20T23:42:56.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-20T23:42:56.108 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.108 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.108 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.109 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.497 INFO:teuthology.orchestra.run.smithi137.stdout:240518168585 2024-09-20T23:42:56.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-20T23:42:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.584 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:56.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:56 smithi137 ceph-mon[23698]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:56.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:56 smithi144 ceph-mon[26057]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:58.436 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:58.436 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:42:58.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:42:58 smithi137 ceph-mon[23698]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:42:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:42:58 smithi144 ceph-mon[26057]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:00.637 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:00 smithi137 ceph-mon[23698]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:00.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:00 smithi144 ceph-mon[26057]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:02.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:02 smithi137 ceph-mon[23698]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:02.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:02 smithi144 ceph-mon[26057]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:04.208 INFO:teuthology.orchestra.run.smithi137.stdout:38654705712 2024-09-20T23:43:04.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:04 smithi137 ceph-mon[23698]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:04.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:04 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/663457977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T23:43:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:04 smithi144 ceph-mon[26057]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:04 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/663457977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-20T23:43:05.530 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-09-20T23:43:05.639 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:06.269 INFO:teuthology.orchestra.run.smithi137.stdout:60129542186 2024-09-20T23:43:06.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:06 smithi137 ceph-mon[23698]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:06.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:06 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/384723727' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T23:43:06.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:06 smithi144 ceph-mon[26057]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:06.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:06 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/384723727' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-20T23:43:07.641 INFO:teuthology.orchestra.run.smithi137.stdout:115964117023 2024-09-20T23:43:07.715 INFO:teuthology.orchestra.run.smithi137.stdout:137438953499 2024-09-20T23:43:07.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:07 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1154418788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T23:43:07.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:07 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/764986161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T23:43:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:07 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1154418788' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-20T23:43:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:07 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/764986161' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-20T23:43:08.379 INFO:teuthology.orchestra.run.smithi137.stdout:81604378661 2024-09-20T23:43:08.579 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953496 got 137438953499 for osd.4 2024-09-20T23:43:08.580 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:08.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:08 smithi137 ceph-mon[23698]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:08.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:08 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2767044529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T23:43:08.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:08 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/334387098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T23:43:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:08 smithi144 ceph-mon[26057]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:08 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2767044529' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-20T23:43:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:08 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/334387098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-20T23:43:09.135 INFO:teuthology.orchestra.run.smithi137.stdout:206158430225 2024-09-20T23:43:09.713 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430222 got 206158430225 for osd.6 2024-09-20T23:43:09.713 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:10.102 INFO:teuthology.orchestra.run.smithi137.stdout:240518168588 2024-09-20T23:43:10.767 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378658 got 81604378661 for osd.2 2024-09-20T23:43:10.768 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:10.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:10 smithi137 ceph-mon[23698]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:10.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:10 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1422654957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T23:43:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:10 smithi144 ceph-mon[26057]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:10 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1422654957' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-20T23:43:11.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-09-20T23:43:11.145 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:11.608 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117020 got 115964117023 for osd.3 2024-09-20T23:43:11.608 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:11.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:43:11.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:11 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:43:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:43:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:11 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:43:11.977 INFO:tasks.cephadm.ceph_manager.ceph:need seq 240518168585 got 240518168588 for osd.7 2024-09-20T23:43:11.977 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:12.613 INFO:teuthology.orchestra.run.smithi137.stdout:171798691862 2024-09-20T23:43:12.910 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:12 smithi137 ceph-mon[23698]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:12.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:12 smithi144 ceph-mon[26057]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:13.229 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691858 got 171798691862 for osd.5 2024-09-20T23:43:13.230 DEBUG:teuthology.parallel:result is None 2024-09-20T23:43:13.230 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-20T23:43:13.230 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T23:43:13.441 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:43:13.905 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:13 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3083917433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T23:43:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:13 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3083917433' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-20T23:43:14.824 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:14 smithi137 ceph-mon[23698]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:14 smithi144 ceph-mon[26057]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:15.210 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:15.212 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-20T23:43:15.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:15 smithi137 ceph-mon[23698]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:15.872 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":199,"stamp":"2024-09-20T23:43:13.404181+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49004,"kb_used_data":6380,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912212,"statfs":{"total":767960285184,"available":767910105088,"internally_reserved":0,"allocated":6533120,"data_stored":3571277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002266"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T23:42:25.175441+0000","last_change":"2024-09-20T23:42:25.175441+0000","last_active":"2024-09-20T23:42:25.175441+0000","last_peered":"2024-09-20T23:42:25.175441+0000","last_clean":"2024-09-20T23:42:25.175441+0000","last_became_active":"2024-09-20T23:42:25.175088+0000","last_became_peered":"2024-09-20T23:42:25.175088+0000","last_unstale":"2024-09-20T23:42:25.175441+0000","last_undegraded":"2024-09-20T23:42:25.175441+0000","last_fullsized":"2024-09-20T23:42:25.175441+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T23:40:21.916124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T23:40:21.916124+0000","last_clean_scrub_stamp":"2024-09-20T23:40:21.916124+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:04:24.605642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168589,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41399999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48199999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36299999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59599999999999997}]}]},{"osd":6,"up_from":48,"seq":206158430227,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.42799999999999999}]},{"osd":1,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.36799999999999999}]},{"osd":2,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.502},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.59299999999999997}]},{"osd":3,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.54600000000000004}]},{"osd":4,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.57199999999999995}]},{"osd":5,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.60199999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":1,"up_from":14,"seq":60129542187,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.33900000000000002,"15min":0.32200000000000001},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57199999999999995,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.251},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.36499999999999999}]},{"osd":2,"last update":"Fri Sep 20 23:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40200000000000002,"15min":0.39400000000000002},"min":{"1min":0.29799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.27300000000000002}]},{"osd":3,"last update":"Fri Sep 20 23:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.434,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40200000000000002}]},{"osd":4,"last update":"Fri Sep 20 23:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.63700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.61299999999999999}]},{"osd":5,"last update":"Fri Sep 20 23:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.68100000000000005}]},{"osd":6,"last update":"Fri Sep 20 23:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.66900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63100000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.31},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Sep 20 23:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.32600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.372}]},{"osd":3,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.436,"15min":0.441},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55300000000000005,"5min":0.624,"15min":0.624},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.311}]},{"osd":4,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82099999999999995,"5min":17.170000000000002,"15min":17.170000000000002},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.626},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6304,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.376},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.34699999999999998}]},{"osd":1,"last update":"Fri Sep 20 23:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.27600000000000002}]},{"osd":3,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61299999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.36099999999999999}]},{"osd":4,"last update":"Fri Sep 20 23:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53800000000000003}]},{"osd":5,"last update":"Fri Sep 20 23:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.60099999999999998}]},{"osd":6,"last update":"Fri Sep 20 23:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.51400000000000001},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":16.620999999999999,"5min":16.620999999999999,"15min":16.620999999999999},"last":0.54300000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50700000000000001}]}]},{"osd":3,"up_from":27,"seq":115964117025,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36899999999999999,"15min":0.36499999999999999},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.26400000000000001}]},{"osd":1,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.27400000000000002}]},{"osd":2,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.439,"15min":0.435},"min":{"1min":0.32400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.28100000000000003}]},{"osd":4,"last update":"Fri Sep 20 23:42:10 2024","interfaces":[{"interface":"back","average":{"1min":0.92700000000000005,"5min":0.92700000000000005,"15min":0.92700000000000005},"min":{"1min":0.42299999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":8.1539999999999999,"5min":8.1539999999999999,"15min":8.1539999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"min":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":8.1829999999999998,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.61499999999999999}]},{"osd":5,"last update":"Fri Sep 20 23:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.60799999999999998}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.54700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57699999999999996}]}]},{"osd":4,"up_from":32,"seq":137438953500,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5792,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739360,"statfs":{"total":95995035648,"available":95989104640,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.76000000000000001,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.37}]},{"osd":1,"last update":"Fri Sep 20 23:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.81299999999999994,"5min":0.81299999999999994,"15min":0.81299999999999994},"last":0.436},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.46000000000000002}]},{"osd":2,"last update":"Fri Sep 20 23:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.56399999999999995,"15min":0.56399999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.76600000000000001,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.44800000000000001}]},{"osd":3,"last update":"Fri Sep 20 23:42:08 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.47999999999999998}]},{"osd":5,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.498}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.51100000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.628}]},{"osd":2,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Sep 20 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T23:43:15.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph pg dump --format=json 2024-09-20T23:43:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:15 smithi144 ceph-mon[26057]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:16.080 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:43:16.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:16 smithi137 ceph-mon[23698]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:16 smithi144 ceph-mon[26057]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:17.916 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:17.917 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-20T23:43:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":201,"stamp":"2024-09-20T23:43:17.404840+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":49004,"kb_used_data":6380,"kb_used_omap":0,"kb_used_meta":42560,"kb_avail":749912212,"statfs":{"total":767960285184,"available":767910105088,"internally_reserved":0,"allocated":6533120,"data_stored":3571277,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43581440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002039"},"pg_stats":[{"pgid":"1.0","version":"23'117","reported_seq":222,"reported_epoch":60,"state":"active+clean","last_fresh":"2024-09-20T23:42:25.175441+0000","last_change":"2024-09-20T23:42:25.175441+0000","last_active":"2024-09-20T23:42:25.175441+0000","last_peered":"2024-09-20T23:42:25.175441+0000","last_clean":"2024-09-20T23:42:25.175441+0000","last_became_active":"2024-09-20T23:42:25.175088+0000","last_became_peered":"2024-09-20T23:42:25.175088+0000","last_unstale":"2024-09-20T23:42:25.175441+0000","last_undegraded":"2024-09-20T23:42:25.175441+0000","last_fullsized":"2024-09-20T23:42:25.175441+0000","mapping_epoch":59,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":60,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-20T23:40:21.916124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-20T23:40:21.916124+0000","last_clean_scrub_stamp":"2024-09-20T23:40:21.916124+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-22T07:04:24.605642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":56,"seq":240518168590,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6060,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739092,"statfs":{"total":95995035648,"available":95988830208,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35699999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54800000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.504}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58399999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54200000000000004}]}]},{"osd":6,"up_from":48,"seq":206158430228,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6188,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93738964,"statfs":{"total":95995035648,"available":95988699136,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68999999999999995,"5min":0.68999999999999995,"15min":0.68999999999999995},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67000000000000004,"5min":0.67000000000000004,"15min":0.67000000000000004},"last":0.58199999999999996}]},{"osd":1,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.56699999999999995,"15min":0.56699999999999995},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.48299999999999998}]},{"osd":2,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.499}]},{"osd":3,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.40999999999999998}]},{"osd":4,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.51200000000000001}]},{"osd":5,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.83899999999999997,"5min":0.83899999999999997,"15min":0.83899999999999997},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.55900000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6372,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:59 2024","interfaces":[{"interface":"back","average":{"1min":0.36899999999999999,"5min":0.33900000000000002,"15min":0.32200000000000001},"min":{"1min":0.216,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.57199999999999995,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.313},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.36499999999999999,"15min":0.34499999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56100000000000005,"5min":0.64400000000000002,"15min":0.64400000000000002},"last":0.38200000000000001}]},{"osd":2,"last update":"Fri Sep 20 23:42:26 2024","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.378,"15min":0.375},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.40200000000000002,"15min":0.39400000000000002},"min":{"1min":0.29799999999999999,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.61699999999999999,"5min":0.65400000000000003,"15min":0.65400000000000003},"last":0.34999999999999998}]},{"osd":3,"last update":"Fri Sep 20 23:42:50 2024","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47999999999999998,"15min":0.48499999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.64800000000000002,"5min":0.65700000000000003,"15min":0.65700000000000003},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.434,"15min":0.437},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40600000000000003}]},{"osd":4,"last update":"Fri Sep 20 23:42:13 2024","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.58199999999999996,"15min":0.58199999999999996},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.502},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.73299999999999998,"5min":0.73299999999999998,"15min":0.73299999999999998},"last":0.57799999999999996}]},{"osd":5,"last update":"Fri Sep 20 23:42:34 2024","interfaces":[{"interface":"back","average":{"1min":0.625,"5min":0.625,"15min":0.625},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.622,"5min":0.622,"15min":0.622},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.81100000000000005,"5min":0.81100000000000005,"15min":0.81100000000000005},"last":0.47999999999999998}]},{"osd":6,"last update":"Fri Sep 20 23:42:57 2024","interfaces":[{"interface":"back","average":{"1min":0.629,"5min":0.629,"15min":0.629},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.49299999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6892,"kb_used_data":1124,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738260,"statfs":{"total":95995035648,"available":95987978240,"internally_reserved":0,"allocated":1150976,"data_stored":777089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.33000000000000002,"15min":0.31},"min":{"1min":0.21099999999999999,"5min":0.21099999999999999,"15min":0.21099999999999999},"max":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.35099999999999998,"15min":0.32900000000000001},"min":{"1min":0.223,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.39400000000000002}]},{"osd":2,"last update":"Fri Sep 20 23:42:24 2024","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.40000000000000002,"15min":0.39200000000000002},"min":{"1min":0.32600000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.432,"5min":0.40000000000000002,"15min":0.39400000000000002},"min":{"1min":0.249,"5min":0.222,"15min":0.222},"max":{"1min":0.54200000000000004,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.372}]},{"osd":3,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.436,"15min":0.441},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.55300000000000005,"5min":0.624,"15min":0.624},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.432,"5min":0.47099999999999997,"15min":0.47699999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.57599999999999996,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.311}]},{"osd":4,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":1.179,"15min":1.2749999999999999},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.82099999999999995,"5min":17.170000000000002,"15min":17.170000000000002},"last":0.68000000000000005}]},{"osd":5,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56599999999999995,"15min":0.56599999999999995},"min":{"1min":0.432,"5min":0.432,"15min":0.432},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.76200000000000001}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.42899999999999999,"5min":0.42899999999999999,"15min":0.42899999999999999},"max":{"1min":0.83599999999999997,"5min":0.83599999999999997,"15min":0.83599999999999997},"last":0.626},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.70999999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72999999999999998}]}]},{"osd":2,"up_from":19,"seq":81604378663,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6304,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738848,"statfs":{"total":95995035648,"available":95988580352,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.38,"15min":0.376},"min":{"1min":0.255,"5min":0.255,"15min":0.255},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.315},{"interface":"front","average":{"1min":0.436,"5min":0.38300000000000001,"15min":0.374},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.254}]},{"osd":1,"last update":"Fri Sep 20 23:42:29 2024","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.39900000000000002,"15min":0.39300000000000002},"min":{"1min":0.28399999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.39500000000000002,"15min":0.39200000000000002},"min":{"1min":0.32700000000000001,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.27500000000000002}]},{"osd":3,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.45500000000000002,"15min":0.45700000000000002},"min":{"1min":0.308,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.55000000000000004,"5min":0.59499999999999997,"15min":0.59499999999999997},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.61299999999999999,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.33900000000000002}]},{"osd":4,"last update":"Fri Sep 20 23:43:08 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.53700000000000003,"15min":0.52800000000000002},"min":{"1min":0.44,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.53800000000000003,"15min":0.53200000000000003},"min":{"1min":0.42399999999999999,"5min":0.372,"15min":0.372},"max":{"1min":0.70899999999999996,"5min":0.70899999999999996,"15min":0.70899999999999996},"last":0.53200000000000003}]},{"osd":5,"last update":"Fri Sep 20 23:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55700000000000005,"5min":0.55700000000000005,"15min":0.55700000000000005},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"min":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"max":{"1min":0.747,"5min":0.747,"15min":0.747},"last":0.66100000000000003}]},{"osd":6,"last update":"Fri Sep 20 23:42:56 2024","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.83799999999999997,"5min":0.83799999999999997,"15min":0.83799999999999997},"last":0.58099999999999996},{"interface":"front","average":{"1min":1.4419999999999999,"5min":1.4419999999999999,"15min":1.4419999999999999},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":16.620999999999999,"5min":16.620999999999999,"15min":16.620999999999999},"last":0.54600000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65600000000000003}]}]},{"osd":3,"up_from":27,"seq":115964117026,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5728,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739424,"statfs":{"total":95995035648,"available":95989170176,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.38900000000000001,"5min":0.36899999999999999,"15min":0.36499999999999999},"min":{"1min":0.27800000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.246},{"interface":"front","average":{"1min":0.433,"5min":0.41699999999999998,"15min":0.41499999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.32600000000000001}]},{"osd":1,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.41099999999999998,"15min":0.40600000000000003},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.43099999999999999,"15min":0.433},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.33900000000000002}]},{"osd":2,"last update":"Fri Sep 20 23:42:48 2024","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.441,"15min":0.436},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.59599999999999997,"5min":0.60999999999999999,"15min":0.60999999999999999},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.439,"15min":0.435},"min":{"1min":0.32400000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.315}]},{"osd":4,"last update":"Fri Sep 20 23:43:14 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.85999999999999999,"15min":0.90400000000000003},"min":{"1min":0.42899999999999999,"5min":0.42299999999999999,"15min":0.42299999999999999},"max":{"1min":0.80100000000000005,"5min":8.1539999999999999,"15min":8.1539999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.64500000000000002,"5min":0.85299999999999998,"15min":0.88800000000000001},"min":{"1min":0.47699999999999998,"5min":0.45400000000000001,"15min":0.45400000000000001},"max":{"1min":0.84499999999999997,"5min":8.1829999999999998,"15min":8.1829999999999998},"last":0.41099999999999998}]},{"osd":5,"last update":"Fri Sep 20 23:42:33 2024","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.435}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.45800000000000002,"5min":0.45800000000000002,"15min":0.45800000000000002},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"max":{"1min":0.81799999999999995,"5min":0.81799999999999995,"15min":0.81799999999999995},"last":0.52000000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47499999999999998}]}]},{"osd":4,"up_from":32,"seq":137438953501,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5792,"kb_used_data":600,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739360,"statfs":{"total":95995035648,"available":95989104640,"internally_reserved":0,"allocated":614400,"data_stored":247896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.26800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.71699999999999997,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.69699999999999995}]},{"osd":1,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.84699999999999998,"5min":0.61099999999999999,"15min":0.57199999999999995},"min":{"1min":0.316,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":6.2050000000000001,"5min":6.2050000000000001,"15min":6.2050000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.52900000000000003,"15min":0.52100000000000002},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79500000000000004,"5min":0.79500000000000004,"15min":0.79500000000000004},"last":0.65800000000000003}]},{"osd":2,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.89000000000000001,"5min":0.629,"15min":0.58499999999999996},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":6.2380000000000004,"5min":6.2380000000000004,"15min":6.2380000000000004},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54700000000000004,"15min":0.54500000000000004},"min":{"1min":0.36699999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.755,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.63600000000000001}]},{"osd":3,"last update":"Fri Sep 20 23:43:11 2024","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.56899999999999995,"15min":0.57199999999999995},"min":{"1min":0.41599999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.67900000000000005,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.53400000000000003,"15min":0.53700000000000003},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.68300000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.68300000000000005}]},{"osd":5,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.66800000000000004},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.46500000000000002,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.39800000000000002}]},{"osd":6,"last update":"Fri Sep 20 23:42:58 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.47999999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]}]},{"osd":5,"up_from":40,"seq":171798691863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5668,"kb_used_data":604,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739484,"statfs":{"total":95995035648,"available":95989231616,"internally_reserved":0,"allocated":618496,"data_stored":248161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.53000000000000003,"15min":0.53000000000000003},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.628}]},{"osd":2,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.66900000000000004}]},{"osd":3,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.59999999999999998}]},{"osd":4,"last update":"Fri Sep 20 23:42:35 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"last":0.46999999999999997}]},{"osd":6,"last update":"Fri Sep 20 23:43:00 2024","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.65800000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55200000000000005}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-20T23:43:18.555 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-20T23:43:18.555 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-20T23:43:18.555 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-20T23:43:18.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph health --format=json 2024-09-20T23:43:18.763 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/mon.smithi137/config 2024-09-20T23:43:18.884 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:18 smithi137 ceph-mon[23698]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:18.884 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:18 smithi137 ceph-mon[23698]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:18 smithi144 ceph-mon[26057]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:18 smithi144 ceph-mon[26057]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:20.486 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:20.486 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-20T23:43:20.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:20 smithi137 ceph-mon[23698]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:20.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:20 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1160505104' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T23:43:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:20 smithi144 ceph-mon[26057]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:20 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1160505104' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-20T23:43:21.150 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-20T23:43:21.150 INFO:tasks.cephadm:Setup complete, yielding 2024-09-20T23:43:21.150 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T23:43:21.159 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:43:21.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph orch status' 2024-09-20T23:43:22.838 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:22 smithi137 ceph-mon[23698]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:22.925 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:22 smithi144 ceph-mon[26057]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:23.298 INFO:teuthology.orchestra.run.smithi137.stdout:Backend: cephadm 2024-09-20T23:43:23.298 INFO:teuthology.orchestra.run.smithi137.stdout:Available: Yes 2024-09-20T23:43:23.298 INFO:teuthology.orchestra.run.smithi137.stdout:Paused: No 2024-09-20T23:43:23.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:23.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:23.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:23.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:23 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:23.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ps' 2024-09-20T23:43:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:23 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:25.461 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:25 smithi137 ceph-mon[23698]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:25.462 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:25 smithi137 ceph-mon[23698]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:25 smithi144 ceph-mon[26057]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:25 smithi144 ceph-mon[26057]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:25.880 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:43:25.880 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (4m) 2m ago 5m 15.5M - ba2b418f427c 40f3a9c71ccc 2024-09-20T23:43:25.880 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (5m) 2m ago 5m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:43:25.880 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (5m) 66s ago 5m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:43:25.880 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (4m) 2m ago 5m 41.2M - 8.3.5 dad864ee21e9 6e1e86bc96e6 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (6m) 2m ago 6m 447M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (5m) 66s ago 5m 412M - 17.2.0 e1d6a67b021e 26154ff41500 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (6m) 2m ago 7m 41.2M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (4m) 66s ago 4m 42.4M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (5m) 2m ago 5m 19.1M - 1dbe0e931976 7f485feb2895 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (4m) 66s ago 4m 17.4M - 1dbe0e931976 8b486a597225 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (4m) 2m ago 4m 33.1M 3216M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (3m) 2m ago 3m 35.7M 3216M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (3m) 2m ago 3m 31.5M 3216M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (2m) 2m ago 2m 12.6M 3216M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:43:25.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (2m) 66s ago 2m 34.9M 3984M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:43:25.882 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (119s) 66s ago 119s 36.2M 3984M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:43:25.882 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (95s) 66s ago 95s 34.0M 3984M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:43:25.882 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (71s) 66s ago 70s 14.3M 3984M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:43:25.882 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (4m) 2m ago 4m 54.4M - 514e6a882f6e 46ab214bac9b 2024-09-20T23:43:26.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph orch ls' 2024-09-20T23:43:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:27 smithi137 ceph-mon[23698]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:27.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:27 smithi137 ceph-mon[23698]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:27.495 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:27 smithi144 ceph-mon[26057]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:27.496 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:27 smithi144 ceph-mon[26057]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:28.447 INFO:teuthology.orchestra.run.smithi137.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-09-20T23:43:28.447 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-09-20T23:43:28.447 INFO:teuthology.orchestra.run.smithi137.stdout:crash 2/2 2m ago 6m * 2024-09-20T23:43:28.447 INFO:teuthology.orchestra.run.smithi137.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-09-20T23:43:28.447 INFO:teuthology.orchestra.run.smithi137.stdout:mgr 2/2 2m ago 6m count:2 2024-09-20T23:43:28.448 INFO:teuthology.orchestra.run.smithi137.stdout:mon 2/2 2m ago 5m smithi137:172.21.15.137=smithi137;smithi144:172.21.15.144=smithi144;count:2 2024-09-20T23:43:28.448 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-09-20T23:43:28.448 INFO:teuthology.orchestra.run.smithi137.stdout:osd 8 2m ago - 2024-09-20T23:43:28.448 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-09-20T23:43:29.057 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph orch host ls' 2024-09-20T23:43:29.351 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:29 smithi137 ceph-mon[23698]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:29.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:29 smithi144 ceph-mon[26057]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:30.540 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:30 smithi137 ceph-mon[23698]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:30.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:30 smithi144 ceph-mon[26057]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:30.950 INFO:teuthology.orchestra.run.smithi137.stdout:HOST ADDR LABELS STATUS 2024-09-20T23:43:30.951 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137 172.21.15.137 2024-09-20T23:43:30.951 INFO:teuthology.orchestra.run.smithi137.stdout:smithi144 172.21.15.144 2024-09-20T23:43:30.951 INFO:teuthology.orchestra.run.smithi137.stdout:2 hosts in cluster 2024-09-20T23:43:31.514 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:31 smithi137 ceph-mon[23698]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:31.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph orch device ls' 2024-09-20T23:43:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:31 smithi144 ceph-mon[26057]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:32.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:32 smithi137 ceph-mon[23698]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:32.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:32 smithi144 ceph-mon[26057]: from='client.14468 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:33.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:33 smithi137 ceph-mon[23698]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:33.542 INFO:teuthology.orchestra.run.smithi137.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-09-20T23:43:33.542 INFO:teuthology.orchestra.run.smithi137.stdout:smithi137 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6233004A400BGN 400G LVM detected, locked 2024-09-20T23:43:33.542 INFO:teuthology.orchestra.run.smithi137.stdout:smithi144 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62050085400BGN 400G LVM detected, locked 2024-09-20T23:43:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:33 smithi144 ceph-mon[26057]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:34.210 INFO:teuthology.run_tasks:Running task vip.exec... 2024-09-20T23:43:34.221 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:43:34.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-09-20T23:43:34.248 INFO:teuthology.orchestra.run.smithi137.stderr:+ systemctl stop nfs-server 2024-09-20T23:43:34.256 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi144.front.sepia.ceph.com 2024-09-20T23:43:34.256 DEBUG:teuthology.orchestra.run.smithi144:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-09-20T23:43:34.286 INFO:teuthology.orchestra.run.smithi144.stderr:+ systemctl stop nfs-server 2024-09-20T23:43:34.295 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T23:43:34.373 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:43:34.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph fs volume create foofs' 2024-09-20T23:43:35.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:35 smithi137 ceph-mon[23698]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:35.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:35 smithi137 ceph-mon[23698]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:35 smithi144 ceph-mon[26057]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:35 smithi144 ceph-mon[26057]: from='client.14472 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:37 smithi137 ceph-mon[23698]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:37 smithi137 ceph-mon[23698]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:37 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-09-20T23:43:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:37 smithi144 ceph-mon[26057]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:37 smithi144 ceph-mon[26057]: from='client.14476 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:37 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-09-20T23:43:38.625 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:38 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-09-20T23:43:38.625 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:38 smithi137 ceph-mon[23698]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T23:43:38.625 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:38 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-09-20T23:43:38.625 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:38 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[23694]: 2024-09-20T23:43:38.247+0000 7fb18182a700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:43:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:38 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-09-20T23:43:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:38 smithi144 ceph-mon[26057]: osdmap e61: 8 total, 8 up, 8 in 2024-09-20T23:43:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:38 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-09-20T23:43:39.009 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-09-20T23:43:39.020 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-09-20T23:43:39.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch ls -f json 2024-09-20T23:43:39.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: pgmap v212: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:39.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-09-20T23:43:39.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T23:43:39.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-09-20T23:43:39.501 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: fsmap foofs:0 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: Saving service mds.foofs spec with placement count:2 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:43:39.502 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T23:43:39.503 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:39 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:39.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: pgmap v212: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: osdmap e62: 8 total, 8 up, 8 in 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: osdmap e63: 8 total, 8 up, 8 in 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: fsmap foofs:0 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: Saving service mds.foofs spec with placement count:2 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T23:43:39.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:39 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:40.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:40 smithi137 ceph-mon[23698]: Deploying daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:43:40.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:40 smithi137 ceph-mon[23698]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T23:43:40.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:40 smithi144 ceph-mon[26057]: Deploying daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:43:40.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:40 smithi144 ceph-mon[26057]: osdmap e64: 8 total, 8 up, 8 in 2024-09-20T23:43:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:41 smithi137 ceph-mon[23698]: pgmap v216: 65 pgs: 11 creating+peering, 2 active+clean, 52 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:41.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:41 smithi137 ceph-mon[23698]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T23:43:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:41 smithi144 ceph-mon[26057]: pgmap v216: 65 pgs: 11 creating+peering, 2 active+clean, 52 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:41 smithi144 ceph-mon[26057]: osdmap e65: 8 total, 8 up, 8 in 2024-09-20T23:43:41.820 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:41.820 INFO:teuthology.orchestra.run.smithi137.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-09-20T23:37:03.647378Z", "last_refresh": "2024-09-20T23:40:46.008141Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-09-20T23:36:58.233914Z", "last_refresh": "2024-09-20T23:40:46.008218Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-09-20T23:37:00.787956Z", "last_refresh": "2024-09-20T23:40:46.008277Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-09-20T23:43:38.269574Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-09-20T23:43:38.264704Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-20T23:36:56.848583Z", "last_refresh": "2024-09-20T23:40:46.008069Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi137:172.21.15.137=smithi137", "smithi144:172.21.15.144=smithi144"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-20T23:37:46.055672Z", "last_refresh": "2024-09-20T23:40:46.007922Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-09-20T23:37:02.534653Z", "last_refresh": "2024-09-20T23:40:46.008334Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-09-20T23:40:46.008448Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-09-20T23:36:59.385712Z", "last_refresh": "2024-09-20T23:40:46.008391Z", "ports": [9095], "running": 1, "size": 1}}] 2024-09-20T23:43:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: pgmap v218: 65 pgs: 1 creating+activating, 11 creating+peering, 26 active+clean, 27 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:42.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:42.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:42.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:43:42.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T23:43:42.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: pgmap v218: 65 pgs: 1 creating+activating, 11 creating+peering, 26 active+clean, 27 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:43:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='client.14480 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:43:42.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-09-20T23:43:42.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:42.854 INFO:tasks.cephadm:mds.foofs has 0/2 2024-09-20T23:43:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: Deploying daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:43:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: mds.? [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] up:boot 2024-09-20T23:43:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: daemon mds.foofs.smithi137.qhqtcp assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-09-20T23:43:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: Cluster is now healthy 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: fsmap foofs:0 1 up:standby 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:creating} 2024-09-20T23:43:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:43 smithi137 ceph-mon[23698]: daemon mds.foofs.smithi137.qhqtcp is now active in filesystem foofs as rank 0 2024-09-20T23:43:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: Deploying daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:43:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: mds.? [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] up:boot 2024-09-20T23:43:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: daemon mds.foofs.smithi137.qhqtcp assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-09-20T23:43:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: Cluster is now healthy 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: fsmap foofs:0 1 up:standby 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:creating} 2024-09-20T23:43:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:43 smithi144 ceph-mon[26057]: daemon mds.foofs.smithi137.qhqtcp is now active in filesystem foofs as rank 0 2024-09-20T23:43:43.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch ls -f json 2024-09-20T23:43:44.549 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:44 smithi144 ceph-mon[26057]: pgmap v219: 65 pgs: 1 creating+activating, 4 creating+peering, 60 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-09-20T23:43:44.549 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:44 smithi144 ceph-mon[26057]: mds.? [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] up:active 2024-09-20T23:43:44.550 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:44 smithi144 ceph-mon[26057]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 2024-09-20T23:43:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:44 smithi137 ceph-mon[23698]: pgmap v219: 65 pgs: 1 creating+activating, 4 creating+peering, 60 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-09-20T23:43:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:44 smithi137 ceph-mon[23698]: mds.? [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] up:active 2024-09-20T23:43:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:44 smithi137 ceph-mon[23698]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 2024-09-20T23:43:45.722 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:45.722 INFO:teuthology.orchestra.run.smithi137.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-09-20T23:37:03.647378Z", "last_refresh": "2024-09-20T23:40:46.008141Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-09-20T23:36:58.233914Z", "last_refresh": "2024-09-20T23:40:46.008218Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-09-20T23:37:00.787956Z", "last_refresh": "2024-09-20T23:40:46.008277Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-09-20T23:43:38.269574Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-09-20T23:43:38.264704Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-20T23:36:56.848583Z", "last_refresh": "2024-09-20T23:40:46.008069Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi137:172.21.15.137=smithi137", "smithi144:172.21.15.144=smithi144"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-20T23:37:46.055672Z", "last_refresh": "2024-09-20T23:40:46.007922Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-09-20T23:37:02.534653Z", "last_refresh": "2024-09-20T23:40:46.008334Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-09-20T23:40:46.008448Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-09-20T23:36:59.385712Z", "last_refresh": "2024-09-20T23:40:46.008391Z", "ports": [9095], "running": 1, "size": 1}}] 2024-09-20T23:43:45.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:45.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:45.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:45.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:45.970 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:boot 2024-09-20T23:43:45.970 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:43:45.970 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:43:45.970 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:45 smithi137 ceph-mon[23698]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:43:46.156 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:46.156 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:46.156 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:46.156 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:46.156 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:boot 2024-09-20T23:43:46.157 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:43:46.157 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:43:46.157 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:45 smithi144 ceph-mon[26057]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:43:46.786 INFO:tasks.cephadm:mds.foofs has 0/2 2024-09-20T23:43:47.115 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:46 smithi137 ceph-mon[23698]: pgmap v220: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-09-20T23:43:47.116 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:46 smithi137 ceph-mon[23698]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:47.116 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:46 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:46 smithi144 ceph-mon[26057]: pgmap v220: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s wr, 1 op/s 2024-09-20T23:43:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:46 smithi144 ceph-mon[26057]: from='client.14488 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:47.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:46 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:47.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch ls -f json 2024-09-20T23:43:48.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:47 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:48.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:47 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:49.127 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:48 smithi137 ceph-mon[23698]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-20T23:43:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:48 smithi144 ceph-mon[26057]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s wr, 5 op/s 2024-09-20T23:43:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:50.402 INFO:teuthology.orchestra.run.smithi137.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-09-20T23:37:03.647378Z", "last_refresh": "2024-09-20T23:40:46.008141Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-09-20T23:36:58.233914Z", "last_refresh": "2024-09-20T23:40:46.008218Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-09-20T23:37:00.787956Z", "last_refresh": "2024-09-20T23:40:46.008277Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-09-20T23:43:38.269574Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-09-20T23:43:38.264704Z", "last_refresh": "2024-09-20T23:43:46.810340Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-20T23:36:56.848583Z", "last_refresh": "2024-09-20T23:40:46.008069Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi137:172.21.15.137=smithi137", "smithi144:172.21.15.144=smithi144"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-20T23:37:46.055672Z", "last_refresh": "2024-09-20T23:40:46.007922Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-09-20T23:37:02.534653Z", "last_refresh": "2024-09-20T23:40:46.008334Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-09-20T23:40:46.008448Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-09-20T23:36:59.385712Z", "last_refresh": "2024-09-20T23:40:46.008391Z", "ports": [9095], "running": 1, "size": 1}}] 2024-09-20T23:43:51.153 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:50 smithi137 ceph-mon[23698]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-20T23:43:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:50 smithi144 ceph-mon[26057]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-09-20T23:43:51.874 INFO:tasks.cephadm:mds.foofs has 1/2 2024-09-20T23:43:52.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:51 smithi137 ceph-mon[23698]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:52.097 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:52.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:51 smithi144 ceph-mon[26057]: from='client.14492 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:52.122 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:52.875 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch ls -f json 2024-09-20T23:43:53.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:53.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:53 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-09-20T23:43:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:43:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:43:53.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:43:53.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:53 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:54.977 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:43:54.977 INFO:teuthology.orchestra.run.smithi137.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-09-20T23:37:03.647378Z", "last_refresh": "2024-09-20T23:43:52.100013Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-09-20T23:36:58.233914Z", "last_refresh": "2024-09-20T23:43:46.809724Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-09-20T23:37:00.787956Z", "last_refresh": "2024-09-20T23:43:52.100145Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-09-20T23:43:38.269574Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-09-20T23:43:38.264704Z", "last_refresh": "2024-09-20T23:43:46.810340Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-20T23:36:56.848583Z", "last_refresh": "2024-09-20T23:43:46.809864Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi137:172.21.15.137=smithi137", "smithi144:172.21.15.144=smithi144"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-20T23:37:46.055672Z", "last_refresh": "2024-09-20T23:43:46.809965Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-09-20T23:37:02.534653Z", "last_refresh": "2024-09-20T23:43:46.810031Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-09-20T23:43:46.810092Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-09-20T23:36:59.385712Z", "last_refresh": "2024-09-20T23:43:52.100282Z", "ports": [9095], "running": 1, "size": 1}}] 2024-09-20T23:43:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:55 smithi137 ceph-mon[23698]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-20T23:43:55.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:55 smithi144 ceph-mon[26057]: pgmap v224: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-09-20T23:43:55.665 INFO:tasks.cephadm:mds.foofs has 2/2 2024-09-20T23:43:55.666 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T23:43:55.681 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:43:55.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-09-20T23:43:56.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:56 smithi137 ceph-mon[23698]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:56.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:56 smithi144 ceph-mon[26057]: from='client.14496 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:43:57.383 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:57 smithi137 ceph-mon[23698]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 3 op/s 2024-09-20T23:43:57.383 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:57 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:57.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:57 smithi144 ceph-mon[26057]: pgmap v225: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 511 B/s wr, 3 op/s 2024-09-20T23:43:57.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:57 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:43:58.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:58 smithi137 ceph-mon[23698]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 2 op/s 2024-09-20T23:43:58.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:58 smithi137 ceph-mon[23698]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:58.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:58 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-09-20T23:43:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:58 smithi144 ceph-mon[26057]: pgmap v226: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 255 B/s wr, 2 op/s 2024-09-20T23:43:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:58 smithi144 ceph-mon[26057]: from='client.14500 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:43:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:58 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-09-20T23:43:59.596 INFO:teuthology.orchestra.run.smithi137.stdout:NFS Cluster Created Successfully 2024-09-20T23:43:59.911 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-09-20T23:43:59.912 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:59 smithi137 ceph-mon[23698]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T23:43:59.912 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:43:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-09-20T23:43:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-09-20T23:43:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:59 smithi144 ceph-mon[26057]: osdmap e66: 8 total, 8 up, 8 in 2024-09-20T23:43:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:43:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-09-20T23:44:00.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: pgmap v228: 97 pgs: 9 creating+peering, 23 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Saving service nfs.foo spec with placement count:2 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.874 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Rados config object exists: conf-nfs.foo 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv-rgw 2024-09-20T23:44:00.875 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:44:00.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T23:44:00.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.876 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:00 smithi137 ceph-mon[23698]: Deploying daemon nfs.foo.0.0.smithi137.gcqarv on smithi137 2024-09-20T23:44:00.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: pgmap v228: 97 pgs: 9 creating+peering, 23 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: osdmap e67: 8 total, 8 up, 8 in 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Saving service nfs.foo spec with placement count:2 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Rados config object exists: conf-nfs.foo 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv-rgw 2024-09-20T23:44:00.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:44:00.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T23:44:00.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:00.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:00 smithi144 ceph-mon[26057]: Deploying daemon nfs.foo.0.0.smithi137.gcqarv on smithi137 2024-09-20T23:44:01.870 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:01 smithi137 ceph-mon[23698]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T23:44:01.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:01 smithi144 ceph-mon[26057]: osdmap e68: 8 total, 8 up, 8 in 2024-09-20T23:44:02.886 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:02 smithi137 ceph-mon[23698]: pgmap v231: 97 pgs: 14 creating+peering, 17 unknown, 66 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:44:02.886 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:02 smithi137 ceph-mon[23698]: mgrmap e20: smithi137.eurecx(active, since 5m), standbys: smithi144.ygsngz 2024-09-20T23:44:02.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:02 smithi144 ceph-mon[26057]: pgmap v231: 97 pgs: 14 creating+peering, 17 unknown, 66 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-09-20T23:44:02.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:02 smithi144 ceph-mon[26057]: mgrmap e20: smithi137.eurecx(active, since 5m), standbys: smithi144.ygsngz 2024-09-20T23:44:03.505 INFO:teuthology.orchestra.run.smithi137.stderr:Invalid command: Unexpected argument '--clusterid' 2024-09-20T23:44:03.505 INFO:teuthology.orchestra.run.smithi137.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-09-20T23:44:03.505 INFO:teuthology.orchestra.run.smithi137.stderr:Error EINVAL: invalid command 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout: "bind": "/fake", 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout: "fs": "foofs", 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout: "path": "/", 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster": "foo", 2024-09-20T23:44:03.858 INFO:teuthology.orchestra.run.smithi137.stdout: "mode": "RW" 2024-09-20T23:44:03.859 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:44:04.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: pgmap v232: 97 pgs: 14 creating+peering, 83 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-09-20T23:44:04.893 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:44:04.894 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:44:04.894 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:04 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: pgmap v232: 97 pgs: 14 creating+peering, 83 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-09-20T23:44:04.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='client.14522 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:44:04.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:04 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:06.875 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:06 smithi144 ceph-mon[26057]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 298 B/s wr, 0 op/s 2024-09-20T23:44:06.927 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:06 smithi137 ceph-mon[23698]: pgmap v233: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 149 B/s rd, 298 B/s wr, 0 op/s 2024-09-20T23:44:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:44:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:44:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: Rados config object exists: conf-nfs.foo 2024-09-20T23:44:07.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb-rgw 2024-09-20T23:44:07.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:44:07.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T23:44:07.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:07.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:07 smithi137 ceph-mon[23698]: Deploying daemon nfs.foo.1.0.smithi144.kqljhb on smithi144 2024-09-20T23:44:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: Rados config object exists: conf-nfs.foo 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb-rgw 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:07 smithi144 ceph-mon[26057]: Deploying daemon nfs.foo.1.0.smithi144.kqljhb on smithi144 2024-09-20T23:44:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:08 smithi144 ceph-mon[26057]: pgmap v234: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-09-20T23:44:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:08 smithi144 ceph-mon[26057]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:09.101 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:08 smithi137 ceph-mon[23698]: pgmap v234: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-09-20T23:44:09.101 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:08 smithi137 ceph-mon[23698]: from='client.14552 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:10.574 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:10 smithi144 ceph-mon[26057]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:10.574 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:10.574 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:10.574 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:10.574 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:10 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:10.616 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:10 smithi137 ceph-mon[23698]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:10.616 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:10.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:10.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:10.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:10 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:11.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:11 smithi137 ceph-mon[23698]: pgmap v235: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T23:44:11.634 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:11 smithi144 ceph-mon[26057]: pgmap v235: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1.5 KiB/s wr, 3 op/s 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:12.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]: dispatch 2024-09-20T23:44:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]: dispatch 2024-09-20T23:44:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]: dispatch 2024-09-20T23:44:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]: dispatch 2024-09-20T23:44:12.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:44:12.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:44:12.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:12.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:13.605 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: pgmap v236: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T23:44:13.606 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:13 smithi137 ceph-mon[23698]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T23:44:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: pgmap v236: 97 pgs: 97 active+clean; 580 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.4 KiB/s wr, 2 op/s 2024-09-20T23:44:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:13.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1c", "id": [7, 4]}]': finished 2024-09-20T23:44:13.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.6", "id": [1, 5]}]': finished 2024-09-20T23:44:13.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [1, 2]}]': finished 2024-09-20T23:44:13.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [1, 2]}]': finished 2024-09-20T23:44:13.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:13 smithi144 ceph-mon[26057]: osdmap e69: 8 total, 8 up, 8 in 2024-09-20T23:44:14.420 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-09-20T23:44:14.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:14.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T23:44:14.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-09-20T23:44:14.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: osdmap e70: 8 total, 8 up, 8 in 2024-09-20T23:44:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: pgmap v239: 97 pgs: 1 peering, 96 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.1 KiB/s wr, 8 op/s 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:14.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:15.075 INFO:teuthology.run_tasks:Running task vip.exec... 2024-09-20T23:44:15.090 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:44:15.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-09-20T23:44:15.118 INFO:teuthology.orchestra.run.smithi137.stderr:+ mkdir /mnt/foo 2024-09-20T23:44:15.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-09-20T23:44:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:++ hostname 2024-09-20T23:44:15.196 INFO:teuthology.orchestra.run.smithi137.stderr:+ mount -t nfs smithi137:/fake /mnt/foo -o sync 2024-09-20T23:44:15.599 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:15 smithi137 ceph-mon[23698]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:15.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-09-20T23:44:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:15 smithi144 ceph-mon[26057]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:15.695 INFO:teuthology.orchestra.run.smithi137.stderr:+ echo test 2024-09-20T23:44:15.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-09-20T23:44:15.769 INFO:teuthology.orchestra.run.smithi137.stderr:+ sync 2024-09-20T23:44:16.021 INFO:teuthology.run_tasks:Running task parallel... 2024-09-20T23:44:16.031 INFO:teuthology.task.parallel:starting parallel... 2024-09-20T23:44:16.032 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-20T23:44:16.032 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-20T23:44:16.033 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:44:16.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-09-20T23:44:16.033 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-09-20T23:44:16.033 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-09-20T23:44:16.034 INFO:teuthology.task.exec:Executing custom commands... 2024-09-20T23:44:16.034 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:44:16.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-09-20T23:44:16.095 INFO:teuthology.orchestra.run.smithi137.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-20T23:44:16.095 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:44:16.095 INFO:teuthology.orchestra.run.smithi137.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-09-20T23:44:16.096 INFO:teuthology.orchestra.run.smithi137.stdout:3 of 5 processes prepared for launch 0 sec 2024-09-20T23:44:16.196 INFO:teuthology.orchestra.run.smithi137.stdout:5 of 5 processes prepared for launch 0 sec 2024-09-20T23:44:16.197 INFO:teuthology.orchestra.run.smithi137.stdout:releasing clients 2024-09-20T23:44:16.313 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:16 smithi137 ceph-mon[23698]: pgmap v240: 97 pgs: 1 peering, 96 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:44:16.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:16 smithi144 ceph-mon[26057]: pgmap v240: 97 pgs: 1 peering, 96 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.7 KiB/s rd, 1.5 KiB/s wr, 5 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:44:17.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 302 55.60 MB/sec warmup 1 sec latency 28.718 ms 2024-09-20T23:44:18.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 617 52.25 MB/sec warmup 2 sec latency 11.883 ms 2024-09-20T23:44:18.626 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:18 smithi137 ceph-mon[23698]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 14 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 159 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:18 smithi144 ceph-mon[26057]: pgmap v241: 97 pgs: 1 peering, 96 active+clean; 14 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 1.8 MiB/s wr, 159 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:19.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 1423 43.36 MB/sec warmup 3 sec latency 13.771 ms 2024-09-20T23:44:20.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2553 36.75 MB/sec warmup 4 sec latency 14.601 ms 2024-09-20T23:44:20.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-09-20T23:44:20.641 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:20 smithi137 ceph-mon[23698]: pgmap v242: 97 pgs: 97 active+clean; 53 MiB data, 74 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 6.8 MiB/s wr, 255 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:20 smithi144 ceph-mon[26057]: pgmap v242: 97 pgs: 97 active+clean; 53 MiB data, 74 MiB used, 715 GiB / 715 GiB avail; 3.6 KiB/s rd, 6.8 MiB/s wr, 255 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:21.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 3705 38.94 MB/sec warmup 5 sec latency 11.481 ms 2024-09-20T23:44:22.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 4427 34.00 MB/sec warmup 6 sec latency 28.733 ms 2024-09-20T23:44:22.651 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:22 smithi137 ceph-mon[23698]: pgmap v243: 97 pgs: 97 active+clean; 88 MiB data, 108 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s wr, 457 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:22.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:22 smithi144 ceph-mon[26057]: pgmap v243: 97 pgs: 97 active+clean; 88 MiB data, 108 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s wr, 457 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:23.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 5469 32.53 MB/sec warmup 7 sec latency 74.272 ms 2024-09-20T23:44:24.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 6245 30.99 MB/sec warmup 8 sec latency 20.731 ms 2024-09-20T23:44:24.725 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:24 smithi137 ceph-mon[23698]: pgmap v244: 97 pgs: 97 active+clean; 131 MiB data, 202 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 15 MiB/s wr, 858 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:24 smithi144 ceph-mon[26057]: pgmap v244: 97 pgs: 97 active+clean; 131 MiB data, 202 MiB used, 715 GiB / 715 GiB avail; 707 B/s rd, 15 MiB/s wr, 858 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:25.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 7288 32.43 MB/sec warmup 9 sec latency 17.577 ms 2024-09-20T23:44:25.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph config set global log_to_journald false --force' 2024-09-20T23:44:26.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 8005 30.03 MB/sec warmup 10 sec latency 19.630 ms 2024-09-20T23:44:26.739 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:26 smithi137 ceph-mon[23698]: pgmap v245: 97 pgs: 97 active+clean; 134 MiB data, 238 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 14 MiB/s wr, 765 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:26.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:26 smithi144 ceph-mon[26057]: pgmap v245: 97 pgs: 97 active+clean; 134 MiB data, 238 MiB used, 715 GiB / 715 GiB avail; 597 B/s rd, 14 MiB/s wr, 765 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:27.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 9031 29.42 MB/sec warmup 11 sec latency 20.496 ms 2024-09-20T23:44:28.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 10070 29.79 MB/sec warmup 12 sec latency 20.260 ms 2024-09-20T23:44:28.808 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:28 smithi137 ceph-mon[23698]: pgmap v246: 97 pgs: 97 active+clean; 146 MiB data, 328 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 17 MiB/s wr, 1.10k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:28 smithi144 ceph-mon[26057]: pgmap v246: 97 pgs: 97 active+clean; 146 MiB data, 328 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 17 MiB/s wr, 1.10k op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-09-20T23:44:29.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 10959 30.04 MB/sec warmup 13 sec latency 18.637 ms 2024-09-20T23:44:30.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 11593 28.30 MB/sec warmup 14 sec latency 23.331 ms 2024-09-20T23:44:30.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:30 smithi137 ceph-mon[23698]: pgmap v247: 97 pgs: 97 active+clean; 144 MiB data, 371 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 17 MiB/s wr, 1.09k op/s 2024-09-20T23:44:30.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:30 smithi144 ceph-mon[26057]: pgmap v247: 97 pgs: 97 active+clean; 144 MiB data, 371 MiB used, 715 GiB / 715 GiB avail; 7.2 KiB/s rd, 17 MiB/s wr, 1.09k op/s 2024-09-20T23:44:31.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 12182 27.45 MB/sec warmup 15 sec latency 32.982 ms 2024-09-20T23:44:31.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph mgr module enable nfs --force' 2024-09-20T23:44:32.197 INFO:teuthology.orchestra.run.smithi137.stdout: 5 13036 26.77 MB/sec warmup 16 sec latency 60.145 ms 2024-09-20T23:44:32.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:32 smithi137 ceph-mon[23698]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-09-20T23:44:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:32 smithi144 ceph-mon[26057]: pgmap v248: 97 pgs: 97 active+clean; 150 MiB data, 414 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 16 MiB/s wr, 1.25k op/s 2024-09-20T23:44:33.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 14059 27.57 MB/sec warmup 17 sec latency 21.795 ms 2024-09-20T23:44:34.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 14719 27.09 MB/sec warmup 18 sec latency 20.230 ms 2024-09-20T23:44:34.597 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:34 smithi137 ceph-mon[23698]: pgmap v249: 97 pgs: 97 active+clean; 155 MiB data, 455 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 MiB/s wr, 1.34k op/s 2024-09-20T23:44:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:34 smithi144 ceph-mon[26057]: pgmap v249: 97 pgs: 97 active+clean; 155 MiB data, 455 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 15 MiB/s wr, 1.34k op/s 2024-09-20T23:44:35.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 15358 26.22 MB/sec warmup 19 sec latency 22.846 ms 2024-09-20T23:44:35.599 INFO:teuthology.orchestra.run.smithi137.stderr:module 'nfs' is already enabled 2024-09-20T23:44:35.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:35 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1803250084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-09-20T23:44:35.602 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:35 smithi137 ceph-mon[23698]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-09-20T23:44:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:35 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1803250084' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-09-20T23:44:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:35 smithi144 ceph-mon[26057]: from='client.? ' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-09-20T23:44:36.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 16492 26.12 MB/sec warmup 20 sec latency 13.395 ms 2024-09-20T23:44:36.605 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:36 smithi137 ceph-mon[23698]: pgmap v250: 97 pgs: 97 active+clean; 156 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 998 op/s 2024-09-20T23:44:36.605 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:36 smithi137 ceph-mon[23698]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-09-20T23:44:36.605 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:36 smithi137 ceph-mon[23698]: mgrmap e21: smithi137.eurecx(active, since 6m), standbys: smithi144.ygsngz 2024-09-20T23:44:36.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-09-20T23:44:36.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:36 smithi144 ceph-mon[26057]: pgmap v250: 97 pgs: 97 active+clean; 156 MiB data, 474 MiB used, 715 GiB / 715 GiB avail; 14 KiB/s rd, 11 MiB/s wr, 998 op/s 2024-09-20T23:44:36.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:36 smithi144 ceph-mon[26057]: from='client.? ' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-09-20T23:44:36.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:36 smithi144 ceph-mon[26057]: mgrmap e21: smithi137.eurecx(active, since 6m), standbys: smithi144.ygsngz 2024-09-20T23:44:37.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 17561 26.85 MB/sec warmup 21 sec latency 17.738 ms 2024-09-20T23:44:38.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 18339 26.54 MB/sec warmup 22 sec latency 16.027 ms 2024-09-20T23:44:38.616 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:38 smithi137 ceph-mon[23698]: pgmap v251: 97 pgs: 97 active+clean; 159 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-09-20T23:44:38.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:38 smithi144 ceph-mon[26057]: pgmap v251: 97 pgs: 97 active+clean; 159 MiB data, 506 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 12 MiB/s wr, 1.12k op/s 2024-09-20T23:44:39.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 19151 26.07 MB/sec warmup 23 sec latency 14.786 ms 2024-09-20T23:44:40.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 20142 25.82 MB/sec warmup 24 sec latency 14.968 ms 2024-09-20T23:44:40.564 INFO:teuthology.orchestra.run.smithi137.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:44:40.626 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:40 smithi137 ceph-mon[23698]: pgmap v252: 97 pgs: 97 active+clean; 171 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-09-20T23:44:40.626 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:40.626 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:40.627 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:40.627 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:40 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:40.900 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:40 smithi144 ceph-mon[26057]: pgmap v252: 97 pgs: 97 active+clean; 171 MiB data, 555 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 12 MiB/s wr, 1.04k op/s 2024-09-20T23:44:40.900 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:40.900 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:44:40.900 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:44:40.900 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:40 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:44:41.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 20598 25.46 MB/sec warmup 25 sec latency 33.993 ms 2024-09-20T23:44:41.634 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-09-20T23:44:41.634 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:44:41.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-09-20T23:44:42.068 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:42 smithi137 ceph-mon[23698]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:42.069 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:42 smithi137 ceph-mon[23698]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:44:42.069 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:42.069 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:42 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:42.069 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:42 smithi137 ceph-mon[23698]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:44:42.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 21109 25.34 MB/sec warmup 26 sec latency 33.114 ms 2024-09-20T23:44:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:42 smithi144 ceph-mon[26057]: from='client.24381 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:42 smithi144 ceph-mon[26057]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:44:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:42 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:44:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:42 smithi144 ceph-mon[26057]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-20T23:44:43.078 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:43 smithi137 ceph-mon[23698]: pgmap v253: 97 pgs: 97 active+clean; 178 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-09-20T23:44:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 21719 25.12 MB/sec warmup 27 sec latency 34.200 ms 2024-09-20T23:44:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:43 smithi144 ceph-mon[26057]: pgmap v253: 97 pgs: 97 active+clean; 178 MiB data, 600 MiB used, 715 GiB / 715 GiB avail; 21 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-09-20T23:44:44.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 22212 24.47 MB/sec warmup 28 sec latency 26.465 ms 2024-09-20T23:44:45.106 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:45 smithi137 ceph-mon[23698]: pgmap v254: 97 pgs: 97 active+clean; 185 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 13 MiB/s wr, 1.15k op/s 2024-09-20T23:44:45.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 22635 23.90 MB/sec warmup 29 sec latency 31.615 ms 2024-09-20T23:44:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:45 smithi144 ceph-mon[26057]: pgmap v254: 97 pgs: 97 active+clean; 185 MiB data, 642 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 13 MiB/s wr, 1.15k op/s 2024-09-20T23:44:46.013 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:44:46.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 23036 23.45 MB/sec warmup 30 sec latency 43.771 ms 2024-09-20T23:44:47.107 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:44:47.107 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (5m) 33s ago 7m 24.7M - ba2b418f427c 40f3a9c71ccc 2024-09-20T23:44:47.107 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (7m) 33s ago 7m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:44:47.107 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (6m) 35s ago 6m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:44:47.107 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (5m) 33s ago 6m 52.3M - 8.3.5 dad864ee21e9 6e1e86bc96e6 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (65s) 33s ago 64s 15.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (62s) 35s ago 62s 12.0M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (8m) 33s ago 8m 460M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (6m) 35s ago 6m 413M - 17.2.0 e1d6a67b021e 26154ff41500 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (8m) 33s ago 8m 44.2M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (6m) 35s ago 6m 40.4M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (43s) 33s ago 43s 51.4M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (38s) 35s ago 37s 48.7M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 33s ago 6m 21.0M - 1dbe0e931976 7f485feb2895 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (6m) 35s ago 6m 17.7M - 1dbe0e931976 8b486a597225 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (5m) 33s ago 5m 43.2M 3216M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (4m) 33s ago 4m 43.7M 3216M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (4m) 33s ago 4m 41.4M 3216M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:44:47.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (4m) 33s ago 4m 39.6M 3216M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:44:47.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (3m) 35s ago 3m 40.1M 3984M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:44:47.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (3m) 35s ago 3m 42.7M 3984M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:44:47.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (2m) 35s ago 2m 42.1M 3984M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:44:47.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (2m) 35s ago 2m 42.5M 3984M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:44:47.109 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (5m) 33s ago 5m 60.9M - 514e6a882f6e 46ab214bac9b 2024-09-20T23:44:47.121 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:47 smithi137 ceph-mon[23698]: pgmap v255: 97 pgs: 97 active+clean; 184 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 917 op/s 2024-09-20T23:44:47.121 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:47 smithi137 ceph-mon[23698]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:47.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 23572 23.06 MB/sec warmup 31 sec latency 26.615 ms 2024-09-20T23:44:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:47 smithi144 ceph-mon[26057]: pgmap v255: 97 pgs: 97 active+clean; 184 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 917 op/s 2024-09-20T23:44:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:47 smithi144 ceph-mon[26057]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:44:47.663 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:44:47.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:44:47.664 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:44:47.664 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-09-20T23:44:47.664 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:44:47.664 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:44:48.094 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736", 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "", 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 image" 2024-09-20T23:44:48.095 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:44:48.156 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:48 smithi137 ceph-mon[23698]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.156 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:48 smithi137 ceph-mon[23698]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.157 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:48 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2142816442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:44:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 23987 22.66 MB/sec warmup 32 sec latency 51.020 ms 2024-09-20T23:44:48.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:48 smithi144 ceph-mon[26057]: from='client.14608 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:48 smithi144 ceph-mon[26057]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:48.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:48 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2142816442' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:44:48.605 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:44:49.198 INFO:teuthology.orchestra.run.smithi137.stdout: 5 24429 22.59 MB/sec warmup 33 sec latency 67.517 ms 2024-09-20T23:44:49.210 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:49 smithi137 ceph-mon[23698]: pgmap v256: 97 pgs: 97 active+clean; 185 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-09-20T23:44:49.210 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:49 smithi137 ceph-mon[23698]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:49.210 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:49 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/2626603574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:44:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:49 smithi144 ceph-mon[26057]: pgmap v256: 97 pgs: 97 active+clean; 185 MiB data, 699 MiB used, 715 GiB / 715 GiB avail; 19 KiB/s rd, 11 MiB/s wr, 995 op/s 2024-09-20T23:44:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:49 smithi144 ceph-mon[26057]: from='client.14618 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:44:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:49 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/2626603574' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:44:50.199 INFO:teuthology.orchestra.run.smithi137.stdout: 5 24831 22.29 MB/sec warmup 34 sec latency 45.913 ms 2024-09-20T23:44:51.199 INFO:teuthology.orchestra.run.smithi137.stdout: 5 25209 22.05 MB/sec warmup 35 sec latency 57.887 ms 2024-09-20T23:44:51.258 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:51 smithi137 ceph-mon[23698]: pgmap v257: 97 pgs: 97 active+clean; 186 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 992 op/s 2024-09-20T23:44:51.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:51 smithi144 ceph-mon[26057]: pgmap v257: 97 pgs: 97 active+clean; 186 MiB data, 703 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 10 MiB/s wr, 992 op/s 2024-09-20T23:44:52.199 INFO:teuthology.orchestra.run.smithi137.stdout: 5 25454 21.56 MB/sec warmup 36 sec latency 86.445 ms 2024-09-20T23:44:53.201 INFO:teuthology.orchestra.run.smithi137.stdout: 5 25921 21.16 MB/sec warmup 37 sec latency 41.771 ms 2024-09-20T23:44:53.305 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:53 smithi137 ceph-mon[23698]: pgmap v258: 97 pgs: 97 active+clean; 189 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 822 op/s 2024-09-20T23:44:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:53 smithi144 ceph-mon[26057]: pgmap v258: 97 pgs: 97 active+clean; 189 MiB data, 737 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.4 MiB/s wr, 822 op/s 2024-09-20T23:44:54.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 26739 21.13 MB/sec warmup 38 sec latency 20.276 ms 2024-09-20T23:44:55.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 27532 21.04 MB/sec warmup 39 sec latency 17.688 ms 2024-09-20T23:44:55.315 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:55 smithi137 ceph-mon[23698]: pgmap v259: 97 pgs: 97 active+clean; 195 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 649 op/s 2024-09-20T23:44:55.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:55 smithi144 ceph-mon[26057]: pgmap v259: 97 pgs: 97 active+clean; 195 MiB data, 773 MiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.3 MiB/s wr, 649 op/s 2024-09-20T23:44:56.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 28248 21.26 MB/sec warmup 40 sec latency 29.391 ms 2024-09-20T23:44:57.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 28975 21.25 MB/sec warmup 41 sec latency 23.346 ms 2024-09-20T23:44:57.325 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:57 smithi137 ceph-mon[23698]: pgmap v260: 97 pgs: 97 active+clean; 201 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 585 op/s 2024-09-20T23:44:57.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:57 smithi144 ceph-mon[26057]: pgmap v260: 97 pgs: 97 active+clean; 201 MiB data, 793 MiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.2 MiB/s wr, 585 op/s 2024-09-20T23:44:58.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 29821 21.15 MB/sec warmup 42 sec latency 16.416 ms 2024-09-20T23:44:59.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 30712 21.10 MB/sec warmup 43 sec latency 16.862 ms 2024-09-20T23:44:59.335 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:44:59 smithi137 ceph-mon[23698]: pgmap v261: 97 pgs: 97 active+clean; 205 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 695 op/s 2024-09-20T23:44:59.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:44:59 smithi144 ceph-mon[26057]: pgmap v261: 97 pgs: 97 active+clean; 205 MiB data, 839 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 7.2 MiB/s wr, 695 op/s 2024-09-20T23:45:00.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 31499 21.19 MB/sec warmup 44 sec latency 23.308 ms 2024-09-20T23:45:01.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 32094 21.22 MB/sec warmup 45 sec latency 26.691 ms 2024-09-20T23:45:01.352 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:01 smithi137 ceph-mon[23698]: pgmap v262: 97 pgs: 97 active+clean; 208 MiB data, 865 MiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.5 MiB/s wr, 667 op/s 2024-09-20T23:45:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:01 smithi144 ceph-mon[26057]: pgmap v262: 97 pgs: 97 active+clean; 208 MiB data, 865 MiB used, 714 GiB / 715 GiB avail; 7.5 KiB/s rd, 7.5 MiB/s wr, 667 op/s 2024-09-20T23:45:02.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 32621 21.03 MB/sec warmup 46 sec latency 34.965 ms 2024-09-20T23:45:03.137 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:03 smithi137 ceph-mon[23698]: pgmap v263: 97 pgs: 97 active+clean; 216 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 724 op/s 2024-09-20T23:45:03.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 33043 20.73 MB/sec warmup 47 sec latency 124.696 ms 2024-09-20T23:45:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:03 smithi144 ceph-mon[26057]: pgmap v263: 97 pgs: 97 active+clean; 216 MiB data, 892 MiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 8.2 MiB/s wr, 724 op/s 2024-09-20T23:45:04.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 33813 20.72 MB/sec warmup 48 sec latency 22.256 ms 2024-09-20T23:45:05.147 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:05 smithi137 ceph-mon[23698]: pgmap v264: 97 pgs: 97 active+clean; 223 MiB data, 935 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 858 op/s 2024-09-20T23:45:05.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 34370 20.49 MB/sec warmup 49 sec latency 50.609 ms 2024-09-20T23:45:05.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:05 smithi144 ceph-mon[26057]: pgmap v264: 97 pgs: 97 active+clean; 223 MiB data, 935 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 858 op/s 2024-09-20T23:45:06.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 34625 20.24 MB/sec warmup 50 sec latency 66.710 ms 2024-09-20T23:45:07.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 34907 20.07 MB/sec warmup 51 sec latency 65.018 ms 2024-09-20T23:45:07.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:07 smithi137 ceph-mon[23698]: pgmap v265: 97 pgs: 97 active+clean; 223 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 852 op/s 2024-09-20T23:45:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:07 smithi144 ceph-mon[26057]: pgmap v265: 97 pgs: 97 active+clean; 223 MiB data, 945 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 852 op/s 2024-09-20T23:45:08.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 35203 19.93 MB/sec warmup 52 sec latency 63.660 ms 2024-09-20T23:45:09.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 35565 19.79 MB/sec warmup 53 sec latency 69.729 ms 2024-09-20T23:45:09.241 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:09 smithi137 ceph-mon[23698]: pgmap v266: 97 pgs: 97 active+clean; 222 MiB data, 990 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 827 op/s 2024-09-20T23:45:09.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:09 smithi144 ceph-mon[26057]: pgmap v266: 97 pgs: 97 active+clean; 222 MiB data, 990 MiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 9.4 MiB/s wr, 827 op/s 2024-09-20T23:45:10.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 36026 19.67 MB/sec warmup 54 sec latency 98.166 ms 2024-09-20T23:45:11.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 36380 19.40 MB/sec warmup 55 sec latency 87.183 ms 2024-09-20T23:45:11.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:11 smithi137 ceph-mon[23698]: pgmap v267: 97 pgs: 97 active+clean; 224 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 812 op/s 2024-09-20T23:45:11.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:11 smithi144 ceph-mon[26057]: pgmap v267: 97 pgs: 97 active+clean; 224 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.0 MiB/s wr, 812 op/s 2024-09-20T23:45:12.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 36693 19.11 MB/sec warmup 56 sec latency 61.900 ms 2024-09-20T23:45:12.318 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:45:12.318 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:12 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:45:12.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:45:12.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:12 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:45:13.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 36928 18.88 MB/sec warmup 57 sec latency 87.091 ms 2024-09-20T23:45:13.349 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:13 smithi137 ceph-mon[23698]: pgmap v268: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 805 op/s 2024-09-20T23:45:13.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:13 smithi144 ceph-mon[26057]: pgmap v268: 97 pgs: 97 active+clean; 229 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.3 MiB/s wr, 805 op/s 2024-09-20T23:45:14.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 37345 18.79 MB/sec warmup 58 sec latency 46.406 ms 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: Upgrade: Target is version 19.1.1-367-g9e2ac70b (unknown) 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5'] 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi137.eurecx) 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: pgmap v269: 97 pgs: 97 active+clean; 236 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.6 MiB/s wr, 730 op/s 2024-09-20T23:45:14.617 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:14 smithi137 ceph-mon[23698]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 on smithi144 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: Upgrade: Target is version 19.1.1-367-g9e2ac70b (unknown) 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, digests ['quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5'] 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi137.eurecx) 2024-09-20T23:45:14.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: pgmap v269: 97 pgs: 97 active+clean; 236 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.9 KiB/s rd, 7.6 MiB/s wr, 730 op/s 2024-09-20T23:45:14.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:14 smithi144 ceph-mon[26057]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 on smithi144 2024-09-20T23:45:15.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 38120 18.71 MB/sec warmup 59 sec latency 75.912 ms 2024-09-20T23:45:16.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 38907 18.95 MB/sec warmup 60 sec latency 45.413 ms 2024-09-20T23:45:16.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:16 smithi137 ceph-mon[23698]: pgmap v270: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 520 op/s 2024-09-20T23:45:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:16 smithi144 ceph-mon[26057]: pgmap v270: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 520 op/s 2024-09-20T23:45:17.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 39316 18.85 MB/sec warmup 61 sec latency 45.779 ms 2024-09-20T23:45:18.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 39599 18.68 MB/sec warmup 62 sec latency 36.582 ms 2024-09-20T23:45:18.646 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:18 smithi137 ceph-mon[23698]: pgmap v271: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 689 op/s 2024-09-20T23:45:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:18 smithi144 ceph-mon[26057]: pgmap v271: 97 pgs: 97 active+clean; 240 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 6.8 MiB/s wr, 689 op/s 2024-09-20T23:45:19.146 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:45:19.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 40179 18.50 MB/sec warmup 63 sec latency 40.370 ms 2024-09-20T23:45:19.650 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:19 smithi137 ceph-mon[23698]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:19 smithi144 ceph-mon[26057]: from='client.24405 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (6m) 66s ago 7m 24.7M - ba2b418f427c 40f3a9c71ccc 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (7m) 66s ago 7m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (7m) 68s ago 6m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (5m) 66s ago 7m 52.3M - 8.3.5 dad864ee21e9 6e1e86bc96e6 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (98s) 66s ago 97s 15.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (95s) 68s ago 95s 12.0M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (8m) 66s ago 8m 460M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:45:20.180 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (6m) 68s ago 6m 413M - 17.2.0 e1d6a67b021e 26154ff41500 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (8m) 66s ago 9m 44.2M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (6m) 68s ago 6m 40.4M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (76s) 66s ago 76s 51.4M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (71s) 68s ago 70s 48.7M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 66s ago 7m 21.0M - 1dbe0e931976 7f485feb2895 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (6m) 68s ago 6m 17.7M - 1dbe0e931976 8b486a597225 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (6m) 66s ago 6m 43.2M 3216M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (5m) 66s ago 5m 43.7M 3216M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (5m) 66s ago 5m 41.4M 3216M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (4m) 66s ago 4m 39.6M 3216M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (4m) 68s ago 4m 40.1M 3984M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (3m) 68s ago 3m 42.7M 3984M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (3m) 68s ago 3m 42.1M 3984M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:45:20.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (3m) 68s ago 3m 42.5M 3984M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:45:20.182 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (6m) 66s ago 6m 60.9M - 514e6a882f6e 46ab214bac9b 2024-09-20T23:45:20.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 40648 18.38 MB/sec warmup 64 sec latency 18.982 ms 2024-09-20T23:45:20.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:20 smithi137 ceph-mon[23698]: pgmap v272: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-09-20T23:45:20.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:20 smithi137 ceph-mon[23698]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:20 smithi137 ceph-mon[23698]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:45:20.842 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:45:20.843 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:20.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:20.844 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:45:20.844 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-09-20T23:45:20.844 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:45:20.844 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:45:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:20 smithi144 ceph-mon[26057]: pgmap v272: 97 pgs: 97 active+clean; 246 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 6.5 KiB/s rd, 7.0 MiB/s wr, 696 op/s 2024-09-20T23:45:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:20 smithi144 ceph-mon[26057]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:20 smithi144 ceph-mon[26057]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:21.202 INFO:teuthology.orchestra.run.smithi137.stdout: 5 41474 18.43 MB/sec warmup 65 sec latency 25.937 ms 2024-09-20T23:45:21.387 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "0/23 daemons upgraded", 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 image on host smithi144" 2024-09-20T23:45:21.388 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:45:21.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:21 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1612720492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:45:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:21 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1612720492' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:45:22.034 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:45:22.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 42231 18.56 MB/sec warmup 66 sec latency 19.948 ms 2024-09-20T23:45:22.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:22 smithi137 ceph-mon[23698]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:22.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:22 smithi137 ceph-mon[23698]: pgmap v273: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 684 op/s 2024-09-20T23:45:22.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:22 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3393159265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:45:22.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:22 smithi144 ceph-mon[26057]: from='client.24421 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:22.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:22 smithi144 ceph-mon[26057]: pgmap v273: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 684 op/s 2024-09-20T23:45:22.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:22 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3393159265' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:45:23.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 43169 18.75 MB/sec warmup 67 sec latency 19.955 ms 2024-09-20T23:45:24.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 43878 18.58 MB/sec warmup 68 sec latency 18.219 ms 2024-09-20T23:45:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:24 smithi137 ceph-mon[23698]: pgmap v274: 97 pgs: 97 active+clean; 247 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.2 MiB/s wr, 681 op/s 2024-09-20T23:45:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:24 smithi144 ceph-mon[26057]: pgmap v274: 97 pgs: 97 active+clean; 247 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.3 KiB/s rd, 7.2 MiB/s wr, 681 op/s 2024-09-20T23:45:25.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 44597 18.64 MB/sec warmup 69 sec latency 28.982 ms 2024-09-20T23:45:26.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 45436 18.66 MB/sec warmup 70 sec latency 26.888 ms 2024-09-20T23:45:26.688 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:26 smithi137 ceph-mon[23698]: pgmap v275: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.0 MiB/s wr, 670 op/s 2024-09-20T23:45:26.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:26 smithi144 ceph-mon[26057]: pgmap v275: 97 pgs: 97 active+clean; 253 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.0 MiB/s wr, 670 op/s 2024-09-20T23:45:27.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 46450 18.94 MB/sec warmup 71 sec latency 16.813 ms 2024-09-20T23:45:28.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 47260 18.84 MB/sec warmup 72 sec latency 19.488 ms 2024-09-20T23:45:28.699 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:28 smithi137 ceph-mon[23698]: pgmap v276: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.6 MiB/s wr, 907 op/s 2024-09-20T23:45:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:28 smithi144 ceph-mon[26057]: pgmap v276: 97 pgs: 97 active+clean; 265 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 15 KiB/s rd, 9.6 MiB/s wr, 907 op/s 2024-09-20T23:45:29.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 48142 18.93 MB/sec warmup 73 sec latency 17.435 ms 2024-09-20T23:45:30.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 48997 18.94 MB/sec warmup 74 sec latency 25.959 ms 2024-09-20T23:45:30.721 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:30 smithi137 ceph-mon[23698]: pgmap v277: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-09-20T23:45:30.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:30 smithi144 ceph-mon[26057]: pgmap v277: 97 pgs: 97 active+clean; 275 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.03k op/s 2024-09-20T23:45:31.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 49658 19.01 MB/sec warmup 75 sec latency 54.386 ms 2024-09-20T23:45:32.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 50349 19.06 MB/sec warmup 76 sec latency 41.772 ms 2024-09-20T23:45:32.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:32 smithi137 ceph-mon[23698]: pgmap v278: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-09-20T23:45:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:32 smithi144 ceph-mon[26057]: pgmap v278: 97 pgs: 97 active+clean; 279 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-09-20T23:45:33.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 51003 18.93 MB/sec warmup 77 sec latency 62.932 ms 2024-09-20T23:45:34.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 51938 19.02 MB/sec warmup 78 sec latency 24.692 ms 2024-09-20T23:45:34.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:34 smithi137 ceph-mon[23698]: pgmap v279: 97 pgs: 97 active+clean; 281 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-09-20T23:45:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:34 smithi144 ceph-mon[26057]: pgmap v279: 97 pgs: 97 active+clean; 281 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-09-20T23:45:35.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 52703 19.03 MB/sec warmup 79 sec latency 24.175 ms 2024-09-20T23:45:36.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 53511 19.17 MB/sec warmup 80 sec latency 24.984 ms 2024-09-20T23:45:36.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:36 smithi137 ceph-mon[23698]: pgmap v280: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-09-20T23:45:36.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:36 smithi144 ceph-mon[26057]: pgmap v280: 97 pgs: 97 active+clean; 282 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-09-20T23:45:37.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 54118 19.08 MB/sec warmup 81 sec latency 23.673 ms 2024-09-20T23:45:38.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 54419 18.89 MB/sec warmup 82 sec latency 81.908 ms 2024-09-20T23:45:38.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:38 smithi137 ceph-mon[23698]: pgmap v281: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-09-20T23:45:38.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:38 smithi144 ceph-mon[26057]: pgmap v281: 97 pgs: 97 active+clean; 290 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.16k op/s 2024-09-20T23:45:39.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 54588 18.72 MB/sec warmup 83 sec latency 74.563 ms 2024-09-20T23:45:40.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 54956 18.60 MB/sec warmup 84 sec latency 90.974 ms 2024-09-20T23:45:40.743 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:40 smithi137 ceph-mon[23698]: pgmap v282: 97 pgs: 97 active+clean; 293 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-09-20T23:45:40.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:40 smithi144 ceph-mon[26057]: pgmap v282: 97 pgs: 97 active+clean; 293 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.06k op/s 2024-09-20T23:45:41.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 55478 18.55 MB/sec warmup 85 sec latency 41.156 ms 2024-09-20T23:45:42.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 56024 18.50 MB/sec warmup 86 sec latency 40.125 ms 2024-09-20T23:45:42.803 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:42 smithi137 ceph-mon[23698]: pgmap v283: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 780 op/s 2024-09-20T23:45:42.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:42 smithi144 ceph-mon[26057]: pgmap v283: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.2 MiB/s wr, 780 op/s 2024-09-20T23:45:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 56508 18.49 MB/sec warmup 87 sec latency 80.522 ms 2024-09-20T23:45:44.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 57062 18.51 MB/sec warmup 88 sec latency 23.909 ms 2024-09-20T23:45:44.844 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:44 smithi137 ceph-mon[23698]: pgmap v284: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 788 op/s 2024-09-20T23:45:44.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:44 smithi144 ceph-mon[26057]: pgmap v284: 97 pgs: 97 active+clean; 291 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.0 MiB/s wr, 788 op/s 2024-09-20T23:45:45.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 57436 18.42 MB/sec warmup 89 sec latency 50.546 ms 2024-09-20T23:45:46.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 57746 18.25 MB/sec warmup 90 sec latency 67.898 ms 2024-09-20T23:45:47.129 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: pgmap v285: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 647 op/s 2024-09-20T23:45:47.129 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: Upgrade: Updating mgr.smithi144.ygsngz 2024-09-20T23:45:47.130 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:47.130 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:45:47.130 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:45:47.130 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:45:47.130 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:47 smithi137 ceph-mon[23698]: Deploying daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:45:47.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 58113 18.12 MB/sec warmup 91 sec latency 73.169 ms 2024-09-20T23:45:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: pgmap v285: 97 pgs: 97 active+clean; 294 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 7.3 MiB/s wr, 647 op/s 2024-09-20T23:45:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: Upgrade: Updating mgr.smithi144.ygsngz 2024-09-20T23:45:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:45:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:45:47.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:45:47.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:47 smithi144 ceph-mon[26057]: Deploying daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:45:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 58521 18.06 MB/sec warmup 92 sec latency 49.916 ms 2024-09-20T23:45:49.145 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:49 smithi144 ceph-mon[26057]: pgmap v286: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 754 op/s 2024-09-20T23:45:49.198 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:49 smithi137 ceph-mon[23698]: pgmap v286: 97 pgs: 97 active+clean; 302 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 14 KiB/s rd, 8.2 MiB/s wr, 754 op/s 2024-09-20T23:45:49.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 58947 17.98 MB/sec warmup 93 sec latency 54.034 ms 2024-09-20T23:45:50.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 59428 17.91 MB/sec warmup 94 sec latency 56.396 ms 2024-09-20T23:45:51.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 59733 17.79 MB/sec warmup 95 sec latency 67.077 ms 2024-09-20T23:45:51.364 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:51 smithi144 ceph-mon[26057]: pgmap v287: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 617 op/s 2024-09-20T23:45:51.364 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:51.364 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:45:51.364 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:45:51.364 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:51 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:45:51.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:51 smithi137 ceph-mon[23698]: pgmap v287: 97 pgs: 97 active+clean; 303 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.9 MiB/s wr, 617 op/s 2024-09-20T23:45:51.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:51.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:45:51.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:45:51.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:51 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:45:52.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 60490 17.91 MB/sec warmup 96 sec latency 53.714 ms 2024-09-20T23:45:52.508 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:45:52.545 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:52 smithi137 ceph-mon[23698]: pgmap v288: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-09-20T23:45:52.650 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:52 smithi144 ceph-mon[26057]: pgmap v288: 97 pgs: 97 active+clean; 306 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-09-20T23:45:53.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 60798 17.83 MB/sec warmup 97 sec latency 60.173 ms 2024-09-20T23:45:53.244 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:45:53.244 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (6m) 99s ago 8m 24.7M - ba2b418f427c 40f3a9c71ccc 2024-09-20T23:45:53.244 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (8m) 99s ago 8m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (7m) 101s ago 7m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (6m) 99s ago 7m 52.3M - 8.3.5 dad864ee21e9 6e1e86bc96e6 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (2m) 99s ago 2m 15.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (2m) 101s ago 2m 12.0M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (9m) 99s ago 9m 460M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 starting - - - - 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (9m) 99s ago 9m 44.2M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (7m) 101s ago 7m 40.4M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (109s) 99s ago 109s 51.4M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (104s) 101s ago 104s 48.7M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 99s ago 7m 21.0M - 1dbe0e931976 7f485feb2895 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (7m) 101s ago 7m 17.7M - 1dbe0e931976 8b486a597225 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (6m) 99s ago 6m 43.2M 3216M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:45:53.245 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (6m) 99s ago 6m 43.7M 3216M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (5m) 99s ago 5m 41.4M 3216M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (5m) 99s ago 5m 39.6M 3216M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (4m) 101s ago 4m 40.1M 3984M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (4m) 101s ago 4m 42.7M 3984M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (4m) 101s ago 4m 42.1M 3984M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (3m) 101s ago 3m 42.5M 3984M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:45:53.246 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (7m) 99s ago 7m 60.9M - 514e6a882f6e 46ab214bac9b 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:45:53.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:45:53.684 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:45:53.684 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-09-20T23:45:53.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:45:53.684 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:45:53.765 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:53 smithi144 ceph-mon[26057]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:53.765 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:53 smithi144 ceph-mon[26057]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:53.765 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:53 smithi144 ceph-mon[26057]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:53.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:53 smithi137 ceph-mon[23698]: from='client.24433 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:53.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:53 smithi137 ceph-mon[23698]: from='client.14654 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:53.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:53 smithi137 ceph-mon[23698]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:54.178 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading mgr daemons" 2024-09-20T23:45:54.179 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:45:54.203 INFO:teuthology.orchestra.run.smithi137.stdout: 5 61018 17.71 MB/sec warmup 98 sec latency 96.126 ms 2024-09-20T23:45:54.624 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:45:54.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:54 smithi137 ceph-mon[23698]: pgmap v289: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.2 MiB/s wr, 599 op/s 2024-09-20T23:45:54.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:54 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3505968888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:45:54.837 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:54 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:54.838 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:54 smithi137 ceph-mon[23698]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:54.851 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:54 smithi144 ceph-mon[26057]: pgmap v289: 97 pgs: 97 active+clean; 308 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.1 KiB/s rd, 6.2 MiB/s wr, 599 op/s 2024-09-20T23:45:54.851 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:54 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3505968888' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:45:54.852 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:54 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:54.852 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:54 smithi144 ceph-mon[26057]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:45:55.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 61289 17.56 MB/sec warmup 99 sec latency 61.689 ms 2024-09-20T23:45:55.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:55 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:55.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:55 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/110003559' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:45:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:55 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:55 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/110003559' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:45:56.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 61596 17.44 MB/sec warmup 100 sec latency 90.452 ms 2024-09-20T23:45:56.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: pgmap v290: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.2 MiB/s wr, 591 op/s 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:45:56.670 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:45:56.673 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: pgmap v290: 97 pgs: 97 active+clean; 313 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.9 KiB/s rd, 6.2 MiB/s wr, 591 op/s 2024-09-20T23:45:56.673 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:45:56.673 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:45:56.673 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:45:56.673 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:45:56.674 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:45:56.674 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:45:57.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 62282 17.45 MB/sec warmup 101 sec latency 31.857 ms 2024-09-20T23:45:57.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:57 smithi137 ceph-mon[23698]: mgrmap e22: smithi137.eurecx(active, since 7m), standbys: smithi144.ygsngz 2024-09-20T23:45:57.877 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:57 smithi144 ceph-mon[26057]: mgrmap e22: smithi137.eurecx(active, since 7m), standbys: smithi144.ygsngz 2024-09-20T23:45:58.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 63029 17.45 MB/sec warmup 102 sec latency 33.176 ms 2024-09-20T23:45:58.688 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:58 smithi137 ceph-mon[23698]: pgmap v291: 97 pgs: 97 active+clean; 313 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 592 op/s 2024-09-20T23:45:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:58 smithi144 ceph-mon[26057]: pgmap v291: 97 pgs: 97 active+clean; 313 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 10 KiB/s rd, 6.3 MiB/s wr, 592 op/s 2024-09-20T23:45:59.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 63632 17.47 MB/sec warmup 103 sec latency 65.260 ms 2024-09-20T23:45:59.705 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi137.eurecx) 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: Failing over to other MGR 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr fail", "who": "smithi137.eurecx"}]: dispatch 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T23:45:59.706 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:45:59 smithi137 ceph-mon[23698]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:46:00.181 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:46:00.181 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi137.eurecx) 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: Failing over to other MGR 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr fail", "who": "smithi137.eurecx"}]: dispatch 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: osdmap e71: 8 total, 8 up, 8 in 2024-09-20T23:46:00.182 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:45:59 smithi144 ceph-mon[26057]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:46:00.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 64184 17.51 MB/sec warmup 104 sec latency 29.994 ms 2024-09-20T23:46:00.716 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "mgr fail", "who": "smithi137.eurecx"}]': finished 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: mgrmap e23: smithi144.ygsngz(active, starting, since 0.994961s), standbys: smithi137.eurecx 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:46:00.717 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: Manager daemon smithi144.ygsngz is now available 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/trash_purge_schedule"}]: dispatch 2024-09-20T23:46:00.718 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:00 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/trash_purge_schedule"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.14206 172.21.15.137:0/3374205884' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "mgr fail", "who": "smithi137.eurecx"}]': finished 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: mgrmap e23: smithi144.ygsngz(active, starting, since 0.994961s), standbys: smithi137.eurecx 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:46:01.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: Manager daemon smithi144.ygsngz is now available 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:46:01.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:01.196 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/trash_purge_schedule"}]: dispatch 2024-09-20T23:46:01.196 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:00 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi144.ygsngz/trash_purge_schedule"}]: dispatch 2024-09-20T23:46:01.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 64615 17.44 MB/sec warmup 105 sec latency 31.933 ms 2024-09-20T23:46:01.973 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: Checking for cert/key for grafana.smithi137 2024-09-20T23:46:01.973 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: mgrmap e24: smithi144.ygsngz(active, since 2s), standbys: smithi137.eurecx 2024-09-20T23:46:01.973 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: pgmap v3: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:01.974 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: Deploying cephadm binary to smithi137 2024-09-20T23:46:01.974 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: Deploying cephadm binary to smithi144 2024-09-20T23:46:01.974 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:01.974 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:01 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: Checking for cert/key for grafana.smithi137 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: mgrmap e24: smithi144.ygsngz(active, since 2s), standbys: smithi137.eurecx 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: pgmap v3: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: Deploying cephadm binary to smithi137 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: Deploying cephadm binary to smithi144 2024-09-20T23:46:02.131 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:02.132 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:01 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:02.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 65103 17.33 MB/sec warmup 106 sec latency 31.592 ms 2024-09-20T23:46:02.980 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:02 smithi137 ceph-mon[23698]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:03.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:02 smithi144 ceph-mon[26057]: pgmap v4: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:03.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 65554 17.31 MB/sec warmup 107 sec latency 124.999 ms 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: [20/Sep/2024:23:46:02] ENGINE Bus STARTING 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: [20/Sep/2024:23:46:02] ENGINE Serving on https://172.21.15.144:7150 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: [20/Sep/2024:23:46:02] ENGINE Error in HTTPServer.serve 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Traceback (most recent call last): 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._run(expiration_interval) 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:46:03.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s = self.context.wrap_socket( 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: return self.sslsocket_class._create( 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self.do_handshake() 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._sslobj.do_handshake() 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: mgrmap e25: smithi144.ygsngz(active, since 4s), standbys: smithi137.eurecx 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: [20/Sep/2024:23:46:02] ENGINE Serving on http://172.21.15.144:8765 2024-09-20T23:46:03.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:03 smithi137 ceph-mon[23698]: [20/Sep/2024:23:46:02] ENGINE Bus STARTED 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: [20/Sep/2024:23:46:02] ENGINE Bus STARTING 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: [20/Sep/2024:23:46:02] ENGINE Serving on https://172.21.15.144:7150 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: [20/Sep/2024:23:46:02] ENGINE Error in HTTPServer.serve 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Traceback (most recent call last): 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:46:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._run(expiration_interval) 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s = self.context.wrap_socket( 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: return self.sslsocket_class._create( 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self.do_handshake() 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._sslobj.do_handshake() 2024-09-20T23:46:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:46:04.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: mgrmap e25: smithi144.ygsngz(active, since 4s), standbys: smithi137.eurecx 2024-09-20T23:46:04.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: [20/Sep/2024:23:46:02] ENGINE Serving on http://172.21.15.144:8765 2024-09-20T23:46:04.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:03 smithi144 ceph-mon[26057]: [20/Sep/2024:23:46:02] ENGINE Bus STARTED 2024-09-20T23:46:04.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 66076 17.28 MB/sec warmup 108 sec latency 33.874 ms 2024-09-20T23:46:04.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:04 smithi137 ceph-mon[23698]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:05.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:04 smithi144 ceph-mon[26057]: pgmap v5: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:05.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 66574 17.21 MB/sec warmup 109 sec latency 147.353 ms 2024-09-20T23:46:05.811 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:05 smithi137 ceph-mon[23698]: mgrmap e26: smithi144.ygsngz(active, since 6s), standbys: smithi137.eurecx 2024-09-20T23:46:06.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:05 smithi144 ceph-mon[26057]: mgrmap e26: smithi144.ygsngz(active, since 6s), standbys: smithi137.eurecx 2024-09-20T23:46:06.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 67121 17.22 MB/sec warmup 110 sec latency 23.809 ms 2024-09-20T23:46:06.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:06 smithi137 ceph-mon[23698]: pgmap v6: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:06 smithi144 ceph-mon[26057]: pgmap v6: 97 pgs: 97 active+clean; 329 MiB data, 1.7 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:46:07.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 67627 17.22 MB/sec warmup 111 sec latency 45.217 ms 2024-09-20T23:46:08.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 68106 17.20 MB/sec warmup 112 sec latency 28.954 ms 2024-09-20T23:46:08.596 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:08 smithi137 ceph-mon[23698]: pgmap v7: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 4.8 MiB/s wr, 642 op/s 2024-09-20T23:46:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:08 smithi144 ceph-mon[26057]: pgmap v7: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 38 KiB/s rd, 4.8 MiB/s wr, 642 op/s 2024-09-20T23:46:09.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 68613 17.10 MB/sec warmup 113 sec latency 56.405 ms 2024-09-20T23:46:09.653 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: Standby manager daemon smithi137.eurecx restarted 2024-09-20T23:46:09.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:46:09.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/crt"}]: dispatch 2024-09-20T23:46:09.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:46:09.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/key"}]: dispatch 2024-09-20T23:46:09.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:09 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:46:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: Standby manager daemon smithi137.eurecx restarted 2024-09-20T23:46:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:46:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/crt"}]: dispatch 2024-09-20T23:46:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:46:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/key"}]: dispatch 2024-09-20T23:46:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:09 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/2253834825' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:46:10.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 69139 17.09 MB/sec warmup 114 sec latency 74.618 ms 2024-09-20T23:46:10.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:10 smithi137 ceph-mon[23698]: mgrmap e27: smithi144.ygsngz(active, since 10s), standbys: smithi137.eurecx 2024-09-20T23:46:10.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:10 smithi137 ceph-mon[23698]: pgmap v8: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.8 MiB/s wr, 498 op/s 2024-09-20T23:46:10.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:10 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:10.669 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:10 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:10 smithi144 ceph-mon[26057]: mgrmap e27: smithi144.ygsngz(active, since 10s), standbys: smithi137.eurecx 2024-09-20T23:46:10.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:10 smithi144 ceph-mon[26057]: pgmap v8: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 3.8 MiB/s wr, 498 op/s 2024-09-20T23:46:10.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:10 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:10.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:10 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:11.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 69694 17.06 MB/sec warmup 115 sec latency 43.033 ms 2024-09-20T23:46:12.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 70229 17.01 MB/sec warmup 116 sec latency 33.479 ms 2024-09-20T23:46:12.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi137 to 1936M 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:12 smithi144 ceph-mon[26057]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 6.5 MiB/s wr, 751 op/s 2024-09-20T23:46:12.489 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi137 to 1936M 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:12.490 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:12 smithi137 ceph-mon[23698]: pgmap v9: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 6.5 MiB/s wr, 751 op/s 2024-09-20T23:46:13.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 70687 17.00 MB/sec warmup 117 sec latency 33.068 ms 2024-09-20T23:46:14.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 71227 17.05 MB/sec warmup 118 sec latency 62.206 ms 2024-09-20T23:46:14.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:14 smithi137 ceph-mon[23698]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-09-20T23:46:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:14 smithi144 ceph-mon[26057]: pgmap v10: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-09-20T23:46:15.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 71637 16.99 MB/sec warmup 119 sec latency 53.253 ms 2024-09-20T23:46:16.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:16 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:16 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:16 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:16.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:16 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:16 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:16 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:16 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:16 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:16 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:16.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:16 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:17.066 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:17 smithi137 ceph-mon[23698]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-09-20T23:46:17.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 72493 11.02 MB/sec execute 1 sec latency 51.142 ms 2024-09-20T23:46:17.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:17 smithi144 ceph-mon[26057]: pgmap v11: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.0 MiB/s wr, 686 op/s 2024-09-20T23:46:18.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 73093 12.15 MB/sec execute 2 sec latency 31.117 ms 2024-09-20T23:46:18.599 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:18 smithi137 ceph-mon[23698]: pgmap v12: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 9.1 MiB/s wr, 972 op/s 2024-09-20T23:46:18.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:18 smithi144 ceph-mon[26057]: pgmap v12: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 30 KiB/s rd, 9.1 MiB/s wr, 972 op/s 2024-09-20T23:46:19.204 INFO:teuthology.orchestra.run.smithi137.stdout: 5 73626 12.75 MB/sec execute 3 sec latency 51.937 ms 2024-09-20T23:46:20.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 74030 12.85 MB/sec execute 4 sec latency 60.913 ms 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: pgmap v13: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.145 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: Adjusting osd_memory_target on smithi144 to 2704M 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:46:21.146 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:21 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:46:21.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 74610 14.63 MB/sec execute 5 sec latency 26.023 ms 2024-09-20T23:46:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: pgmap v13: 97 pgs: 97 active+clean; 338 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.3 MiB/s wr, 599 op/s 2024-09-20T23:46:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: Adjusting osd_memory_target on smithi144 to 2704M 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:46:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:21 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:22 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 75178 14.98 MB/sec execute 6 sec latency 30.854 ms 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:46:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:22.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:22 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:23.154 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:23 smithi137 ceph-mon[23698]: Reconfiguring alertmanager.smithi137 (dependencies changed)... 2024-09-20T23:46:23.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:23 smithi137 ceph-mon[23698]: Deploying daemon alertmanager.smithi137 on smithi137 2024-09-20T23:46:23.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:23 smithi137 ceph-mon[23698]: pgmap v14: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 880 op/s 2024-09-20T23:46:23.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 75685 13.64 MB/sec execute 7 sec latency 37.707 ms 2024-09-20T23:46:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:23 smithi144 ceph-mon[26057]: Reconfiguring alertmanager.smithi137 (dependencies changed)... 2024-09-20T23:46:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:23 smithi144 ceph-mon[26057]: Deploying daemon alertmanager.smithi137 on smithi137 2024-09-20T23:46:23.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:23 smithi144 ceph-mon[26057]: pgmap v14: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.2 MiB/s wr, 880 op/s 2024-09-20T23:46:24.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 76177 13.43 MB/sec execute 8 sec latency 27.412 ms 2024-09-20T23:46:25.171 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:25 smithi137 ceph-mon[23698]: pgmap v15: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-09-20T23:46:25.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 76901 14.31 MB/sec execute 9 sec latency 30.532 ms 2024-09-20T23:46:25.235 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:46:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:25 smithi144 ceph-mon[26057]: pgmap v15: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-09-20T23:46:26.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 77518 14.58 MB/sec execute 10 sec latency 44.185 ms 2024-09-20T23:46:26.394 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (7m) 15s ago 8m 26.4M - 0.23.0 ba2b418f427c 40f3a9c71ccc 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (8m) 15s ago 8m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (8m) 11s ago 8m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (7m) 15s ago 8m 51.4M - dad864ee21e9 6e1e86bc96e6 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (2m) 15s ago 2m 46.1M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (2m) 11s ago 2m 12.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (9m) 15s ago 9m 168M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (36s) 11s ago 8m 483M - 19.1.1-367-g9e2ac70b cc9251b744a3 46ac06d87a19 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (9m) 15s ago 10m 50.3M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (7m) 11s ago 7m 43.6M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (2m) 15s ago 2m 257M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (2m) 11s ago 2m 73.8M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (8m) 15s ago 8m 21.5M - 1.3.1 1dbe0e931976 7f485feb2895 2024-09-20T23:46:26.395 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (7m) 11s ago 7m 19.4M - 1.3.1 1dbe0e931976 8b486a597225 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (7m) 15s ago 7m 282M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (6m) 15s ago 6m 215M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (6m) 15s ago 6m 167M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (5m) 15s ago 5m 231M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (5m) 11s ago 5m 252M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (4m) 11s ago 4m 282M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (4m) 11s ago 4m 248M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (4m) 11s ago 4m 349M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:46:26.396 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (7m) 15s ago 7m 67.5M - 2.33.4 514e6a882f6e 46ab214bac9b 2024-09-20T23:46:27.095 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:46:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:46:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:46:27.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:46:27.096 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:46:27.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 78243 15.52 MB/sec execute 11 sec latency 30.702 ms 2024-09-20T23:46:27.295 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:27 smithi137 ceph-mon[23698]: from='client.24483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:27.296 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:27 smithi137 ceph-mon[23698]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-09-20T23:46:27.296 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:27 smithi137 ceph-mon[23698]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:27 smithi144 ceph-mon[26057]: from='client.24483 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:27 smithi144 ceph-mon[26057]: pgmap v16: 97 pgs: 97 active+clean; 333 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 6.0 MiB/s wr, 566 op/s 2024-09-20T23:46:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:27 smithi144 ceph-mon[26057]: from='client.14702 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "", 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:46:27.691 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:46:28.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 78903 15.65 MB/sec execute 12 sec latency 35.430 ms 2024-09-20T23:46:28.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:28 smithi137 ceph-mon[23698]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:28.300 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:28 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1693329030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:46:28.374 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:46:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:28 smithi144 ceph-mon[26057]: from='client.24491 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:28 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1693329030' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:46:29.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 79510 15.18 MB/sec execute 13 sec latency 19.259 ms 2024-09-20T23:46:29.306 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:29 smithi137 ceph-mon[23698]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.306 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:29 smithi137 ceph-mon[23698]: pgmap v17: 97 pgs: 97 active+clean; 327 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 913 op/s 2024-09-20T23:46:29.306 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:29 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3520550615' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:46:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:29 smithi144 ceph-mon[26057]: from='client.24499 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:46:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:29 smithi144 ceph-mon[26057]: pgmap v17: 97 pgs: 97 active+clean; 327 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 KiB/s rd, 9.4 MiB/s wr, 913 op/s 2024-09-20T23:46:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:29 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3520550615' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:46:30.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 80165 15.30 MB/sec execute 14 sec latency 29.592 ms 2024-09-20T23:46:31.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 80753 15.45 MB/sec execute 15 sec latency 34.994 ms 2024-09-20T23:46:31.322 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:31 smithi137 ceph-mon[23698]: pgmap v18: 97 pgs: 97 active+clean; 327 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.3 MiB/s wr, 627 op/s 2024-09-20T23:46:31.323 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:31 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:31.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:31 smithi144 ceph-mon[26057]: pgmap v18: 97 pgs: 97 active+clean; 327 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.3 MiB/s wr, 627 op/s 2024-09-20T23:46:31.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:31 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:32.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 81205 15.38 MB/sec execute 16 sec latency 58.859 ms 2024-09-20T23:46:33.114 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:33 smithi137 ceph-mon[23698]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 998 op/s 2024-09-20T23:46:33.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 81530 15.17 MB/sec execute 17 sec latency 79.520 ms 2024-09-20T23:46:33.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:33 smithi144 ceph-mon[26057]: pgmap v19: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 10 MiB/s wr, 998 op/s 2024-09-20T23:46:34.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 81829 14.82 MB/sec execute 18 sec latency 79.789 ms 2024-09-20T23:46:35.151 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:35 smithi137 ceph-mon[23698]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-09-20T23:46:35.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 82110 14.56 MB/sec execute 19 sec latency 59.503 ms 2024-09-20T23:46:35.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:35 smithi144 ceph-mon[26057]: pgmap v20: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-09-20T23:46:36.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 82423 14.16 MB/sec execute 20 sec latency 50.818 ms 2024-09-20T23:46:37.188 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:37 smithi137 ceph-mon[23698]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-09-20T23:46:37.205 INFO:teuthology.orchestra.run.smithi137.stdout: 5 82755 13.64 MB/sec execute 21 sec latency 78.239 ms 2024-09-20T23:46:37.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:37 smithi144 ceph-mon[26057]: pgmap v21: 97 pgs: 97 active+clean; 325 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.4 MiB/s wr, 718 op/s 2024-09-20T23:46:38.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 83065 13.32 MB/sec execute 22 sec latency 61.687 ms 2024-09-20T23:46:39.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 83373 13.14 MB/sec execute 23 sec latency 64.675 ms 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 928 op/s 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:39 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: pgmap v22: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.9 MiB/s wr, 928 op/s 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:39 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:46:40.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 83607 12.89 MB/sec execute 24 sec latency 74.425 ms 2024-09-20T23:46:40.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:40 smithi137 ceph-mon[23698]: Reconfiguring grafana.smithi137 (dependencies changed)... 2024-09-20T23:46:40.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:40 smithi137 ceph-mon[23698]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T23:46:40.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:40 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:46:40.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:40 smithi137 ceph-mon[23698]: Reconfiguring daemon grafana.smithi137 on smithi137 2024-09-20T23:46:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:40 smithi144 ceph-mon[26057]: Reconfiguring grafana.smithi137 (dependencies changed)... 2024-09-20T23:46:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:40 smithi144 ceph-mon[26057]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-20T23:46:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:40 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-20T23:46:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:40 smithi144 ceph-mon[26057]: Reconfiguring daemon grafana.smithi137 on smithi137 2024-09-20T23:46:41.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 83912 12.63 MB/sec execute 25 sec latency 71.851 ms 2024-09-20T23:46:41.247 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:41 smithi137 ceph-mon[23698]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 581 op/s 2024-09-20T23:46:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:41 smithi144 ceph-mon[26057]: pgmap v23: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.5 MiB/s wr, 581 op/s 2024-09-20T23:46:42.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 84439 12.68 MB/sec execute 26 sec latency 40.237 ms 2024-09-20T23:46:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 84947 12.91 MB/sec execute 27 sec latency 92.412 ms 2024-09-20T23:46:43.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:43 smithi137 ceph-mon[23698]: pgmap v24: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 770 op/s 2024-09-20T23:46:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:43 smithi144 ceph-mon[26057]: pgmap v24: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.0 MiB/s wr, 770 op/s 2024-09-20T23:46:44.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 85514 13.12 MB/sec execute 28 sec latency 67.906 ms 2024-09-20T23:46:44.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:44 smithi137 ceph-mon[23698]: pgmap v25: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s 2024-09-20T23:46:44.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:44 smithi144 ceph-mon[26057]: pgmap v25: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s 2024-09-20T23:46:45.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 85877 13.01 MB/sec execute 29 sec latency 41.872 ms 2024-09-20T23:46:45.542 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:45 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:45 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:46:46.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 86199 12.70 MB/sec execute 30 sec latency 49.480 ms 2024-09-20T23:46:46.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:46 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:46.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:46 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:46.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:46 smithi137 ceph-mon[23698]: Reconfiguring node-exporter.smithi137 (dependencies changed)... 2024-09-20T23:46:46.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:46 smithi137 ceph-mon[23698]: Deploying daemon node-exporter.smithi137 on smithi137 2024-09-20T23:46:46.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:46 smithi137 ceph-mon[23698]: pgmap v26: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s 2024-09-20T23:46:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:46 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:46 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:46 smithi144 ceph-mon[26057]: Reconfiguring node-exporter.smithi137 (dependencies changed)... 2024-09-20T23:46:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:46 smithi144 ceph-mon[26057]: Deploying daemon node-exporter.smithi137 on smithi137 2024-09-20T23:46:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:46 smithi144 ceph-mon[26057]: pgmap v26: 97 pgs: 97 active+clean; 323 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s 2024-09-20T23:46:47.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 86584 12.52 MB/sec execute 31 sec latency 52.920 ms 2024-09-20T23:46:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 87029 12.51 MB/sec execute 32 sec latency 43.561 ms 2024-09-20T23:46:48.859 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:48 smithi137 ceph-mon[23698]: pgmap v27: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.1 MiB/s wr, 661 op/s 2024-09-20T23:46:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:48 smithi144 ceph-mon[26057]: pgmap v27: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.6 KiB/s rd, 7.1 MiB/s wr, 661 op/s 2024-09-20T23:46:49.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 87483 12.57 MB/sec execute 33 sec latency 48.968 ms 2024-09-20T23:46:50.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 87885 12.45 MB/sec execute 34 sec latency 124.494 ms 2024-09-20T23:46:50.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:50 smithi137 ceph-mon[23698]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-09-20T23:46:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:50 smithi144 ceph-mon[26057]: pgmap v28: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.6 MiB/s wr, 451 op/s 2024-09-20T23:46:51.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 88352 12.54 MB/sec execute 35 sec latency 53.951 ms 2024-09-20T23:46:52.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 88881 12.68 MB/sec execute 36 sec latency 34.045 ms 2024-09-20T23:46:53.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:52 smithi137 ceph-mon[23698]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 696 op/s 2024-09-20T23:46:53.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:52 smithi144 ceph-mon[26057]: pgmap v29: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 6.9 MiB/s wr, 696 op/s 2024-09-20T23:46:53.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 89279 12.66 MB/sec execute 37 sec latency 42.805 ms 2024-09-20T23:46:54.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 89686 12.54 MB/sec execute 38 sec latency 66.680 ms 2024-09-20T23:46:54.815 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:54 smithi137 ceph-mon[23698]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-09-20T23:46:55.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:54 smithi144 ceph-mon[26057]: pgmap v30: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-09-20T23:46:55.206 INFO:teuthology.orchestra.run.smithi137.stdout: 5 90081 12.40 MB/sec execute 39 sec latency 38.173 ms 2024-09-20T23:46:56.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 90531 12.40 MB/sec execute 40 sec latency 43.819 ms 2024-09-20T23:46:56.844 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:56 smithi137 ceph-mon[23698]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-09-20T23:46:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:56 smithi144 ceph-mon[26057]: pgmap v31: 97 pgs: 97 active+clean; 333 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-09-20T23:46:57.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 91038 12.48 MB/sec execute 41 sec latency 41.655 ms 2024-09-20T23:46:58.123 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:58 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:58.124 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:58 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:58.124 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:58 smithi137 ceph-mon[23698]: Reconfiguring prometheus.smithi137 (dependencies changed)... 2024-09-20T23:46:58.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 91517 12.42 MB/sec execute 42 sec latency 34.195 ms 2024-09-20T23:46:58.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:58 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:58.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:58 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:46:58.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:58 smithi144 ceph-mon[26057]: Reconfiguring prometheus.smithi137 (dependencies changed)... 2024-09-20T23:46:58.917 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:46:59.135 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:59 smithi137 ceph-mon[23698]: Deploying daemon prometheus.smithi137 on smithi137 2024-09-20T23:46:59.135 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:46:59 smithi137 ceph-mon[23698]: pgmap v32: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 780 op/s 2024-09-20T23:46:59.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 91974 12.46 MB/sec execute 43 sec latency 49.403 ms 2024-09-20T23:46:59.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:59 smithi144 ceph-mon[26057]: Deploying daemon prometheus.smithi137 on smithi137 2024-09-20T23:46:59.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:46:59 smithi144 ceph-mon[26057]: pgmap v32: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.1 MiB/s wr, 780 op/s 2024-09-20T23:47:00.031 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:00.031 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 starting - - - - 2024-09-20T23:47:00.031 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (9m) 49s ago 9m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (8m) 44s ago 8m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 starting - - - - 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (3m) 49s ago 3m 46.1M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (3m) 44s ago 3m 12.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:9283 running (10m) 49s ago 10m 168M - 17.2.0 e1d6a67b021e 66144db97b84 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (70s) 44s ago 8m 483M - 19.1.1-367-g9e2ac70b cc9251b744a3 46ac06d87a19 2024-09-20T23:47:00.032 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (10m) 49s ago 10m 50.3M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (8m) 44s ago 8m 43.6M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (2m) 49s ago 2m 257M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (2m) 44s ago 2m 73.8M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 starting - - - - 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (8m) 44s ago 8m 19.4M - 1.3.1 1dbe0e931976 8b486a597225 2024-09-20T23:47:00.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (7m) 49s ago 7m 282M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (7m) 49s ago 7m 215M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (6m) 49s ago 6m 167M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (6m) 49s ago 6m 231M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (5m) 44s ago 5m 252M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (5m) 44s ago 5m 282M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (5m) 44s ago 5m 248M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:47:00.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (4m) 44s ago 4m 349M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:47:00.035 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (8m) 49s ago 8m 67.5M - 2.33.4 514e6a882f6e 46ab214bac9b 2024-09-20T23:47:00.150 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:00 smithi137 ceph-mon[23698]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:00.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 92320 12.47 MB/sec execute 44 sec latency 33.927 ms 2024-09-20T23:47:00.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:00 smithi144 ceph-mon[26057]: from='client.14720 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:00.761 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:47:00.762 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:47:00.763 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:47:01.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:01 smithi137 ceph-mon[23698]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:01.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:01 smithi137 ceph-mon[23698]: pgmap v33: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 518 op/s 2024-09-20T23:47:01.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:01 smithi137 ceph-mon[23698]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:01.155 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:01 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:01.156 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:01 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/586671445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:47:01.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 92801 12.52 MB/sec execute 45 sec latency 36.830 ms 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "1/23 daemons upgraded", 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "", 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:47:01.389 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:47:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:01 smithi144 ceph-mon[26057]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:01 smithi144 ceph-mon[26057]: pgmap v33: 97 pgs: 97 active+clean; 331 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.1 MiB/s wr, 518 op/s 2024-09-20T23:47:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:01 smithi144 ceph-mon[26057]: from='client.14728 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:01 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:01.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:01 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/586671445' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:47:01.969 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:47:02.175 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:02 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/3951764506' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:47:02.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 93241 12.48 MB/sec execute 46 sec latency 50.355 ms 2024-09-20T23:47:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:02 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/3951764506' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:47:03.196 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:03 smithi137 ceph-mon[23698]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:03.196 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:03 smithi137 ceph-mon[23698]: pgmap v34: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 779 op/s 2024-09-20T23:47:03.207 INFO:teuthology.orchestra.run.smithi137.stdout: 5 93718 12.35 MB/sec execute 47 sec latency 47.939 ms 2024-09-20T23:47:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:03 smithi144 ceph-mon[26057]: from='client.14736 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:03.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:03 smithi144 ceph-mon[26057]: pgmap v34: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 8.0 MiB/s wr, 779 op/s 2024-09-20T23:47:04.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 94075 12.32 MB/sec execute 48 sec latency 51.612 ms 2024-09-20T23:47:05.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 94418 12.24 MB/sec execute 49 sec latency 49.993 ms 2024-09-20T23:47:05.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:05 smithi137 ceph-mon[23698]: pgmap v35: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-09-20T23:47:05.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:05 smithi144 ceph-mon[26057]: pgmap v35: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-09-20T23:47:06.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 94826 12.27 MB/sec execute 50 sec latency 75.939 ms 2024-09-20T23:47:07.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 95405 12.33 MB/sec execute 51 sec latency 81.097 ms 2024-09-20T23:47:07.277 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:07 smithi137 ceph-mon[23698]: pgmap v36: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-09-20T23:47:07.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:07 smithi144 ceph-mon[26057]: pgmap v36: 97 pgs: 97 active+clean; 340 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.8 MiB/s wr, 534 op/s 2024-09-20T23:47:08.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 95741 12.32 MB/sec execute 52 sec latency 49.376 ms 2024-09-20T23:47:08.797 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:08 smithi137 ceph-mon[23698]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 787 op/s 2024-09-20T23:47:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:08 smithi144 ceph-mon[26057]: pgmap v37: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.1 MiB/s wr, 787 op/s 2024-09-20T23:47:09.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 96081 12.27 MB/sec execute 53 sec latency 61.505 ms 2024-09-20T23:47:10.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 96444 12.25 MB/sec execute 54 sec latency 56.825 ms 2024-09-20T23:47:10.821 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:10 smithi137 ceph-mon[23698]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 514 op/s 2024-09-20T23:47:11.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:10 smithi144 ceph-mon[26057]: pgmap v38: 97 pgs: 97 active+clean; 331 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.8 KiB/s rd, 5.3 MiB/s wr, 514 op/s 2024-09-20T23:47:11.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 96807 12.19 MB/sec execute 55 sec latency 61.437 ms 2024-09-20T23:47:12.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 97212 12.05 MB/sec execute 56 sec latency 42.991 ms 2024-09-20T23:47:12.841 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:12 smithi137 ceph-mon[23698]: pgmap v39: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 756 op/s 2024-09-20T23:47:13.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:12 smithi144 ceph-mon[26057]: pgmap v39: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.9 MiB/s wr, 756 op/s 2024-09-20T23:47:13.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 97653 12.05 MB/sec execute 57 sec latency 36.386 ms 2024-09-20T23:47:14.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 98123 12.08 MB/sec execute 58 sec latency 71.900 ms 2024-09-20T23:47:14.882 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:14 smithi137 ceph-mon[23698]: pgmap v40: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-09-20T23:47:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:14 smithi144 ceph-mon[26057]: pgmap v40: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-09-20T23:47:15.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 98620 12.13 MB/sec execute 59 sec latency 64.315 ms 2024-09-20T23:47:15.913 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:15 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:16.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:15 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:16.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 99003 12.10 MB/sec execute 60 sec latency 32.362 ms 2024-09-20T23:47:16.936 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:16 smithi137 ceph-mon[23698]: pgmap v41: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-09-20T23:47:17.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:16 smithi144 ceph-mon[26057]: pgmap v41: 97 pgs: 97 active+clean; 335 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.0 MiB/s wr, 494 op/s 2024-09-20T23:47:17.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 99353 12.08 MB/sec execute 61 sec latency 47.396 ms 2024-09-20T23:47:18.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 99755 12.10 MB/sec execute 62 sec latency 58.366 ms 2024-09-20T23:47:18.725 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:18 smithi137 ceph-mon[23698]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 755 op/s 2024-09-20T23:47:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:18 smithi144 ceph-mon[26057]: pgmap v42: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 755 op/s 2024-09-20T23:47:19.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 100040 12.07 MB/sec execute 63 sec latency 54.750 ms 2024-09-20T23:47:19.909 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:19 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:19.909 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:19 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:19.909 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:19 smithi137 ceph-mon[23698]: Reconfiguring node-exporter.smithi144 (dependencies changed)... 2024-09-20T23:47:19.909 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:19 smithi137 ceph-mon[23698]: Deploying daemon node-exporter.smithi144 on smithi144 2024-09-20T23:47:20.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:19 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:20.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:19 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:20.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:19 smithi144 ceph-mon[26057]: Reconfiguring node-exporter.smithi144 (dependencies changed)... 2024-09-20T23:47:20.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:19 smithi144 ceph-mon[26057]: Deploying daemon node-exporter.smithi144 on smithi144 2024-09-20T23:47:20.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 100381 11.97 MB/sec execute 64 sec latency 84.643 ms 2024-09-20T23:47:20.914 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:20 smithi137 ceph-mon[23698]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-09-20T23:47:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:20 smithi144 ceph-mon[26057]: pgmap v43: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.2 MiB/s wr, 502 op/s 2024-09-20T23:47:21.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 100696 11.85 MB/sec execute 65 sec latency 38.883 ms 2024-09-20T23:47:22.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 100995 11.77 MB/sec execute 66 sec latency 39.791 ms 2024-09-20T23:47:22.935 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:22 smithi137 ceph-mon[23698]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 714 op/s 2024-09-20T23:47:23.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:22 smithi144 ceph-mon[26057]: pgmap v44: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.4 MiB/s wr, 714 op/s 2024-09-20T23:47:23.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 101287 11.70 MB/sec execute 67 sec latency 46.485 ms 2024-09-20T23:47:24.210 INFO:teuthology.orchestra.run.smithi137.stdout: 5 101610 11.70 MB/sec execute 68 sec latency 54.467 ms 2024-09-20T23:47:24.997 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:24.998 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:24 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: pgmap v45: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T23:47:25.050 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:25.051 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:47:25.051 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T23:47:25.051 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:25.051 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:24 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:25.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 101882 11.65 MB/sec execute 69 sec latency 69.419 ms 2024-09-20T23:47:26.036 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:26.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:25 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:26.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-20T23:47:26.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi137.front.sepia.ceph.com:3000"}]: dispatch 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: from='mon.? -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi137.front.sepia.ceph.com:9095"}]: dispatch 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:26.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:25 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:26.208 INFO:teuthology.orchestra.run.smithi137.stdout: 5 102109 11.55 MB/sec execute 70 sec latency 53.784 ms 2024-09-20T23:47:27.046 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: pgmap v46: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-09-20T23:47:27.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:27.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:47:27.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:47:27.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:47:27.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:26 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:47:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: pgmap v46: 97 pgs: 97 active+clean; 337 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.8 MiB/s wr, 473 op/s 2024-09-20T23:47:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:47:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:47:27.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:47:27.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:26 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:47:27.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 102647 11.63 MB/sec execute 71 sec latency 70.448 ms 2024-09-20T23:47:28.062 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:27 smithi137 ceph-mon[23698]: Upgrade: Updating mgr.smithi137.eurecx 2024-09-20T23:47:28.063 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:27 smithi137 ceph-mon[23698]: Deploying daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:47:28.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:27 smithi144 ceph-mon[26057]: Upgrade: Updating mgr.smithi137.eurecx 2024-09-20T23:47:28.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:27 smithi144 ceph-mon[26057]: Deploying daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:47:28.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 103154 11.68 MB/sec execute 72 sec latency 29.000 ms 2024-09-20T23:47:29.067 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:28 smithi137 ceph-mon[23698]: pgmap v47: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.5 MiB/s wr, 652 op/s 2024-09-20T23:47:29.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:28 smithi144 ceph-mon[26057]: pgmap v47: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 9.9 KiB/s rd, 6.5 MiB/s wr, 652 op/s 2024-09-20T23:47:29.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 103675 11.73 MB/sec execute 73 sec latency 37.105 ms 2024-09-20T23:47:30.210 INFO:teuthology.orchestra.run.smithi137.stdout: 5 104098 11.67 MB/sec execute 74 sec latency 55.491 ms 2024-09-20T23:47:31.085 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:30 smithi137 ceph-mon[23698]: pgmap v48: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 391 op/s 2024-09-20T23:47:31.085 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:30 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:30 smithi144 ceph-mon[26057]: pgmap v48: 97 pgs: 97 active+clean; 335 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.0 MiB/s wr, 391 op/s 2024-09-20T23:47:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:30 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:31.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 104455 11.61 MB/sec execute 75 sec latency 40.215 ms 2024-09-20T23:47:32.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 104773 11.56 MB/sec execute 76 sec latency 96.950 ms 2024-09-20T23:47:32.539 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:47:33.116 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:32 smithi137 ceph-mon[23698]: pgmap v49: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 665 op/s 2024-09-20T23:47:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:32 smithi144 ceph-mon[26057]: pgmap v49: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 6.8 MiB/s wr, 665 op/s 2024-09-20T23:47:33.209 INFO:teuthology.orchestra.run.smithi137.stdout: 5 105132 11.57 MB/sec execute 77 sec latency 53.541 ms 2024-09-20T23:47:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:47:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 starting - - - - 2024-09-20T23:47:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (9m) 83s ago 9m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:47:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (9m) 78s ago 9m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:47:33.612 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 starting - - - - 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (3m) 83s ago 3m 46.1M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (3m) 78s ago 3m 12.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 starting - - - - 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (103s) 78s ago 9m 483M - 19.1.1-367-g9e2ac70b cc9251b744a3 46ac06d87a19 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (11m) 83s ago 11m 50.3M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (9m) 78s ago 9m 43.6M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (3m) 83s ago 3m 257M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (3m) 78s ago 3m 73.8M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 starting - - - - 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 starting - - - - 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (8m) 83s ago 8m 282M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (7m) 83s ago 7m 215M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (7m) 83s ago 7m 167M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:47:33.613 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (6m) 83s ago 6m 231M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:47:33.614 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (6m) 78s ago 6m 252M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:47:33.614 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (6m) 78s ago 6m 282M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:47:33.614 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (5m) 78s ago 5m 248M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:47:33.614 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (5m) 78s ago 5m 349M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:47:33.614 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 starting - - - - 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: mgrmap e28: smithi144.ygsngz(active, since 94s), standbys: smithi137.eurecx 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.142 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:33 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.210 INFO:teuthology.orchestra.run.smithi137.stdout: 5 105410 11.54 MB/sec execute 78 sec latency 57.352 ms 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:47:34.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 13, 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:47:34.326 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:47:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='client.14744 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: mgrmap e28: smithi144.ygsngz(active, since 94s), standbys: smithi137.eurecx 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='client.24533 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:33 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "", 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading mgr daemons", 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:47:34.910 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:47:35.160 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:34 smithi137 ceph-mon[23698]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:35.161 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:34 smithi137 ceph-mon[23698]: pgmap v50: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 453 op/s 2024-09-20T23:47:35.161 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:34 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1438984109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:47:35.210 INFO:teuthology.orchestra.run.smithi137.stdout: 5 105773 11.52 MB/sec execute 79 sec latency 78.551 ms 2024-09-20T23:47:35.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:34 smithi144 ceph-mon[26057]: from='client.14756 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:35.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:34 smithi144 ceph-mon[26057]: pgmap v50: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 453 op/s 2024-09-20T23:47:35.281 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:34 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1438984109' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:47:35.611 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:47:36.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:36 smithi137 ceph-mon[23698]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:36.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:36 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/4054012283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:47:36.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:36 smithi144 ceph-mon[26057]: from='client.24539 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:47:36.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:36 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/4054012283' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:47:36.210 INFO:teuthology.orchestra.run.smithi137.stdout: 5 106060 11.46 MB/sec execute 80 sec latency 47.227 ms 2024-09-20T23:47:37.186 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:37 smithi137 ceph-mon[23698]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 453 op/s 2024-09-20T23:47:37.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 106373 11.44 MB/sec execute 81 sec latency 37.343 ms 2024-09-20T23:47:37.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:37 smithi144 ceph-mon[26057]: pgmap v51: 97 pgs: 97 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 7.3 KiB/s rd, 4.5 MiB/s wr, 453 op/s 2024-09-20T23:47:38.211 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:38 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:38.211 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:38 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:38.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 106676 11.41 MB/sec execute 82 sec latency 45.784 ms 2024-09-20T23:47:38.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:38 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:38.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:38 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:39.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 107011 11.38 MB/sec execute 83 sec latency 45.641 ms 2024-09-20T23:47:39.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:39 smithi137 ceph-mon[23698]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.5 MiB/s wr, 645 op/s 2024-09-20T23:47:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:39 smithi144 ceph-mon[26057]: pgmap v52: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.4 KiB/s rd, 6.5 MiB/s wr, 645 op/s 2024-09-20T23:47:40.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 107436 11.40 MB/sec execute 84 sec latency 42.579 ms 2024-09-20T23:47:41.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 107959 11.36 MB/sec execute 85 sec latency 40.685 ms 2024-09-20T23:47:41.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-09-20T23:47:41.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: Standby manager daemon smithi137.eurecx restarted 2024-09-20T23:47:41.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:47:41.269 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/crt"}]: dispatch 2024-09-20T23:47:41.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:47:41.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/key"}]: dispatch 2024-09-20T23:47:41.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:41 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:47:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: pgmap v53: 97 pgs: 97 active+clean; 334 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.7 MiB/s wr, 465 op/s 2024-09-20T23:47:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: Standby manager daemon smithi137.eurecx restarted 2024-09-20T23:47:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: Standby manager daemon smithi137.eurecx started 2024-09-20T23:47:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/crt"}]: dispatch 2024-09-20T23:47:41.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:47:41.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi137.eurecx/key"}]: dispatch 2024-09-20T23:47:41.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:41 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:47:42.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 108901 11.53 MB/sec execute 86 sec latency 34.375 ms 2024-09-20T23:47:42.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:42 smithi137 ceph-mon[23698]: mgrmap e29: smithi144.ygsngz(active, since 102s), standbys: smithi137.eurecx 2024-09-20T23:47:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:42 smithi144 ceph-mon[26057]: mgrmap e29: smithi144.ygsngz(active, since 102s), standbys: smithi137.eurecx 2024-09-20T23:47:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 109571 11.63 MB/sec execute 87 sec latency 41.499 ms 2024-09-20T23:47:43.287 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:43 smithi137 ceph-mon[23698]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.0 MiB/s wr, 679 op/s 2024-09-20T23:47:43.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:43 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:43.288 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:43 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:43 smithi144 ceph-mon[26057]: pgmap v54: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.0 MiB/s wr, 679 op/s 2024-09-20T23:47:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:43 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:43 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:44.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 109907 11.61 MB/sec execute 88 sec latency 37.886 ms 2024-09-20T23:47:44.307 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:44 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:44.308 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:44 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:44.308 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:44 smithi137 ceph-mon[23698]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-09-20T23:47:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:44 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:44 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:44 smithi144 ceph-mon[26057]: pgmap v55: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-09-20T23:47:45.212 INFO:teuthology.orchestra.run.smithi137.stdout: 5 110244 11.58 MB/sec execute 89 sec latency 49.408 ms 2024-09-20T23:47:45.323 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:45 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:45 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:46.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 110559 11.55 MB/sec execute 90 sec latency 97.331 ms 2024-09-20T23:47:46.354 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:46 smithi137 ceph-mon[23698]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-09-20T23:47:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:46 smithi144 ceph-mon[26057]: pgmap v56: 97 pgs: 97 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 5.3 KiB/s rd, 4.2 MiB/s wr, 406 op/s 2024-09-20T23:47:47.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 110888 11.53 MB/sec execute 91 sec latency 53.873 ms 2024-09-20T23:47:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 111252 11.48 MB/sec execute 92 sec latency 43.684 ms 2024-09-20T23:47:48.661 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:48 smithi137 ceph-mon[23698]: pgmap v57: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-09-20T23:47:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:48 smithi144 ceph-mon[26057]: pgmap v57: 97 pgs: 97 active+clean; 342 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 7.7 MiB/s wr, 732 op/s 2024-09-20T23:47:49.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 111709 11.44 MB/sec execute 93 sec latency 33.477 ms 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: Failing over to other MGR 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:49.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:49.992 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:49 smithi137 ceph-mon[23698]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: Upgrade: Need to upgrade myself (mgr.smithi144.ygsngz) 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: Failing over to other MGR 2024-09-20T23:47:50.165 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 172.21.15.144:0/2526083150' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:50.166 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:50.166 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:49 smithi144 ceph-mon[26057]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T23:47:50.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 112003 11.41 MB/sec execute 94 sec latency 46.925 ms 2024-09-20T23:47:50.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd='[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]': finished 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: mgrmap e30: smithi137.eurecx(active, starting, since 0.870941s) 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:47:50.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:47:50.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:50 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:47:51.039 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.24429 ' entity='mgr.smithi144.ygsngz' cmd='[{"prefix": "mgr fail", "who": "smithi144.ygsngz"}]': finished 2024-09-20T23:47:51.039 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: mgrmap e30: smithi137.eurecx(active, starting, since 0.870941s) 2024-09-20T23:47:51.039 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:47:51.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:47:51.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:50 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:47:51.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 112243 11.39 MB/sec execute 95 sec latency 58.524 ms 2024-09-20T23:47:52.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 112634 11.39 MB/sec execute 96 sec latency 63.932 ms 2024-09-20T23:47:52.309 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:52 smithi137 ceph-mon[23698]: mgrmap e31: smithi137.eurecx(active, since 2s) 2024-09-20T23:47:52.310 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:52 smithi137 ceph-mon[23698]: [20/Sep/2024:23:47:51] ENGINE Bus STARTING 2024-09-20T23:47:52.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:52 smithi144 ceph-mon[26057]: mgrmap e31: smithi137.eurecx(active, since 2s) 2024-09-20T23:47:52.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:52 smithi144 ceph-mon[26057]: [20/Sep/2024:23:47:51] ENGINE Bus STARTING 2024-09-20T23:47:53.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 113018 11.39 MB/sec execute 97 sec latency 45.501 ms 2024-09-20T23:47:53.861 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: [20/Sep/2024:23:47:51] ENGINE Serving on http://172.21.15.137:8765 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: pgmap v4: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: [20/Sep/2024:23:47:51] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: [20/Sep/2024:23:47:51] ENGINE Bus STARTED 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: [20/Sep/2024:23:47:51] ENGINE Error in HTTPServer.serve 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Traceback (most recent call last): 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:47:53.862 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._run(expiration_interval) 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s = self.context.wrap_socket( 2024-09-20T23:47:53.863 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: return self.sslsocket_class._create( 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self.do_handshake() 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._sslobj.do_handshake() 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:47:53.864 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:53 smithi137 ceph-mon[23698]: mgrmap e32: smithi137.eurecx(active, since 4s) 2024-09-20T23:47:53.888 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: [20/Sep/2024:23:47:51] ENGINE Serving on http://172.21.15.137:8765 2024-09-20T23:47:53.888 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: pgmap v4: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:53.888 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: [20/Sep/2024:23:47:51] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:47:53.888 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: [20/Sep/2024:23:47:51] ENGINE Bus STARTED 2024-09-20T23:47:53.888 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: [20/Sep/2024:23:47:51] ENGINE Error in HTTPServer.serve 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Traceback (most recent call last): 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._run(expiration_interval) 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s = self.context.wrap_socket( 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:47:53.889 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: return self.sslsocket_class._create( 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self.do_handshake() 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._sslobj.do_handshake() 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:47:53.890 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:53 smithi144 ceph-mon[26057]: mgrmap e32: smithi137.eurecx(active, since 4s) 2024-09-20T23:47:54.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 113354 11.39 MB/sec execute 98 sec latency 58.385 ms 2024-09-20T23:47:55.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 113688 11.34 MB/sec execute 99 sec latency 53.508 ms 2024-09-20T23:47:55.889 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:55 smithi137 ceph-mon[23698]: pgmap v5: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:55.889 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:55 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:55.890 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:55 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:55.890 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:55 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:55.890 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:55 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:56.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:55 smithi144 ceph-mon[26057]: pgmap v5: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:56.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:55 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:56.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:55 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:56.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:55 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:56.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:55 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:56.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 114118 11.35 MB/sec execute 100 sec latency 35.459 ms 2024-09-20T23:47:56.926 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:56 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:47:56.927 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:47:56.927 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:47:56.927 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:47:56.927 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:56 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:47:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:56 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:47:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:47:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:47:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:47:57.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:56 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/1708272712' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:47:57.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 114514 11.36 MB/sec execute 101 sec latency 51.009 ms 2024-09-20T23:47:57.938 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:57 smithi137 ceph-mon[23698]: pgmap v6: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:57.938 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:57 smithi137 ceph-mon[23698]: mgrmap e33: smithi137.eurecx(active, since 7s), standbys: smithi144.ygsngz 2024-09-20T23:47:57.938 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:57 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:57 smithi144 ceph-mon[26057]: pgmap v6: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail 2024-09-20T23:47:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:57 smithi144 ceph-mon[26057]: mgrmap e33: smithi137.eurecx(active, since 7s), standbys: smithi144.ygsngz 2024-09-20T23:47:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:57 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:47:58.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 114981 11.33 MB/sec execute 102 sec latency 39.200 ms 2024-09-20T23:47:59.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 115545 11.33 MB/sec execute 103 sec latency 33.833 ms 2024-09-20T23:47:59.972 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:59 smithi137 ceph-mon[23698]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 4.3 MiB/s wr, 410 op/s 2024-09-20T23:47:59.973 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:59 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:47:59.973 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:47:59 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:00.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:59 smithi144 ceph-mon[26057]: pgmap v7: 97 pgs: 97 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 35 KiB/s rd, 4.3 MiB/s wr, 410 op/s 2024-09-20T23:48:00.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:59 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:00.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:47:59 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:00.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 116227 11.47 MB/sec execute 104 sec latency 36.236 ms 2024-09-20T23:48:01.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 116991 11.56 MB/sec execute 105 sec latency 28.978 ms 2024-09-20T23:48:01.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:01 smithi137 ceph-mon[23698]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 316 op/s 2024-09-20T23:48:01.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:01 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:01.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:01 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:01 smithi144 ceph-mon[26057]: pgmap v8: 97 pgs: 97 active+clean; 343 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 3.3 MiB/s wr, 316 op/s 2024-09-20T23:48:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:01 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:01.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:01 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:02.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 117558 11.59 MB/sec execute 106 sec latency 42.521 ms 2024-09-20T23:48:03.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 118096 11.63 MB/sec execute 107 sec latency 77.234 ms 2024-09-20T23:48:03.748 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:03 smithi137 ceph-mon[23698]: pgmap v9: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-09-20T23:48:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:03 smithi144 ceph-mon[26057]: pgmap v9: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 27 KiB/s rd, 6.4 MiB/s wr, 628 op/s 2024-09-20T23:48:04.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 118598 11.59 MB/sec execute 108 sec latency 41.847 ms 2024-09-20T23:48:05.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 119024 11.58 MB/sec execute 109 sec latency 36.159 ms 2024-09-20T23:48:05.787 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:05 smithi137 ceph-mon[23698]: pgmap v10: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 563 op/s 2024-09-20T23:48:05.787 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:05 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:05.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:05 smithi144 ceph-mon[26057]: pgmap v10: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 563 op/s 2024-09-20T23:48:05.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:05 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:06.164 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:48:06.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 119540 11.63 MB/sec execute 110 sec latency 38.373 ms 2024-09-20T23:48:07.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 120355 11.74 MB/sec execute 111 sec latency 25.411 ms 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (90s) 8s ago 10m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (10m) 8s ago 10m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (9m) 12s ago 9m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (82s) 8s ago 10m 46.6M - dad864ee21e9 c491347d0b92 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (4m) 8s ago 4m 39.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:48:07.348 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (4m) 12s ago 4m 13.4M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (35s) 8s ago 11m 473M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283 running (2m) 12s ago 9m 103M - 19.1.1-367-g9e2ac70b cc9251b744a3 46ac06d87a19 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (11m) 8s ago 11m 56.0M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (9m) 12s ago 9m 41.0M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (4m) 8s ago 4m 263M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (3m) 12s ago 3m 79.7M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (71s) 8s ago 10m 8786k - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (43s) 12s ago 9m 8376k - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (8m) 8s ago 8m 429M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (8m) 8s ago 8m 330M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (7m) 8s ago 7m 260M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (7m) 8s ago 7m 399M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (7m) 12s ago 7m 388M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:48:07.349 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (6m) 12s ago 6m 427M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:48:07.350 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (6m) 12s ago 6m 362M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:48:07.350 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (5m) 12s ago 5m 514M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:48:07.350 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (49s) 8s ago 9m 42.4M - 2.43.0 a07b618ecd1d 2d9c129a4f41 2024-09-20T23:48:07.795 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: pgmap v11: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 563 op/s 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:07.796 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:07 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: pgmap v11: 97 pgs: 97 active+clean; 344 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 5.7 MiB/s wr, 563 op/s 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='client.14792 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='client.14796 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:07.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:07 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:08.103 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:48:08.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:08.105 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:48:08.105 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-09-20T23:48:08.105 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:48:08.105 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:48:08.105 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:48:08.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 120864 11.74 MB/sec execute 112 sec latency 35.718 ms 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "", 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:48:08.729 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/729454344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:08.800 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:08 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: from='client.14800 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:48:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:48:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:08.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:08.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/729454344' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:08.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:08.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:08 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 121416 11.79 MB/sec execute 113 sec latency 23.426 ms 2024-09-20T23:48:09.458 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: pgmap v12: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.9 MiB/s wr, 875 op/s 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.806 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:09 smithi137 ceph-mon[23698]: from='client.? 172.21.15.137:0/1754481610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:48:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: pgmap v12: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 29 KiB/s rd, 8.9 MiB/s wr, 875 op/s 2024-09-20T23:48:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='client.14808 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:09.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:09 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1754481610' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:48:10.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 121965 11.81 MB/sec execute 114 sec latency 28.492 ms 2024-09-20T23:48:10.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:10 smithi137 ceph-mon[23698]: Reconfiguring prometheus.smithi137 (dependencies changed)... 2024-09-20T23:48:10.813 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:10 smithi137 ceph-mon[23698]: Reconfiguring daemon prometheus.smithi137 on smithi137 2024-09-20T23:48:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:10 smithi144 ceph-mon[26057]: Reconfiguring prometheus.smithi137 (dependencies changed)... 2024-09-20T23:48:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:10 smithi144 ceph-mon[26057]: Reconfiguring daemon prometheus.smithi137 on smithi137 2024-09-20T23:48:11.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 122392 11.77 MB/sec execute 115 sec latency 47.984 ms 2024-09-20T23:48:11.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:11 smithi137 ceph-mon[23698]: pgmap v13: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 644 op/s 2024-09-20T23:48:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:11 smithi144 ceph-mon[26057]: pgmap v13: 97 pgs: 97 active+clean; 345 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.5 MiB/s wr, 644 op/s 2024-09-20T23:48:12.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 122824 11.78 MB/sec execute 116 sec latency 41.120 ms 2024-09-20T23:48:13.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 123155 11.78 MB/sec execute 117 sec latency 63.222 ms 2024-09-20T23:48:13.697 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:13 smithi137 ceph-mon[23698]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 982 op/s 2024-09-20T23:48:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:13 smithi144 ceph-mon[26057]: pgmap v14: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 14 KiB/s rd, 10 MiB/s wr, 982 op/s 2024-09-20T23:48:14.213 INFO:teuthology.orchestra.run.smithi137.stdout: 5 123385 11.73 MB/sec execute 118 sec latency 49.354 ms 2024-09-20T23:48:15.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 123639 11.71 MB/sec execute 119 sec latency 65.623 ms 2024-09-20T23:48:15.786 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:15 smithi137 ceph-mon[23698]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-09-20T23:48:15.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:15 smithi144 ceph-mon[26057]: pgmap v15: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-09-20T23:48:16.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 123998 11.70 MB/sec execute 120 sec latency 50.307 ms 2024-09-20T23:48:17.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 124338 11.65 MB/sec execute 121 sec latency 45.686 ms 2024-09-20T23:48:17.858 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:17 smithi137 ceph-mon[23698]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-09-20T23:48:17.859 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:17 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:17.859 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:17 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:17.859 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:17 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:48:17.859 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:17 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:17 smithi144 ceph-mon[26057]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.7 MiB/s wr, 650 op/s 2024-09-20T23:48:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:17 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:17 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:17.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:17 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:48:17.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:17 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:18.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 124658 11.64 MB/sec execute 122 sec latency 63.291 ms 2024-09-20T23:48:18.878 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:18 smithi137 ceph-mon[23698]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:48:18.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:18 smithi144 ceph-mon[26057]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-20T23:48:19.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 124949 11.61 MB/sec execute 123 sec latency 91.989 ms 2024-09-20T23:48:19.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:19 smithi137 ceph-mon[23698]: pgmap v17: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 842 op/s 2024-09-20T23:48:19.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:19 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:48:19.903 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:19 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:48:19.903 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:19 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:19 smithi144 ceph-mon[26057]: pgmap v17: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 13 KiB/s rd, 8.7 MiB/s wr, 842 op/s 2024-09-20T23:48:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:19 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:48:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:19 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:48:19.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:19 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:20.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 125456 11.60 MB/sec execute 124 sec latency 40.479 ms 2024-09-20T23:48:20.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:20 smithi137 ceph-mon[23698]: Upgrade: Updating mgr.smithi144.ygsngz 2024-09-20T23:48:20.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:20 smithi137 ceph-mon[23698]: Deploying daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:48:20.919 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:20 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:20 smithi144 ceph-mon[26057]: Upgrade: Updating mgr.smithi144.ygsngz 2024-09-20T23:48:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:20 smithi144 ceph-mon[26057]: Deploying daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:48:20.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:20 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:21.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 125880 11.58 MB/sec execute 125 sec latency 65.226 ms 2024-09-20T23:48:21.933 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:21 smithi137 ceph-mon[23698]: pgmap v18: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 530 op/s 2024-09-20T23:48:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:21 smithi144 ceph-mon[26057]: pgmap v18: 97 pgs: 97 active+clean; 345 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.5 MiB/s wr, 530 op/s 2024-09-20T23:48:22.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 126252 11.57 MB/sec execute 126 sec latency 67.808 ms 2024-09-20T23:48:23.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 126618 11.57 MB/sec execute 127 sec latency 48.370 ms 2024-09-20T23:48:23.700 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:23 smithi144 ceph-mon[26057]: pgmap v19: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-09-20T23:48:23.737 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:23 smithi137 ceph-mon[23698]: pgmap v19: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.8 MiB/s wr, 766 op/s 2024-09-20T23:48:24.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 126961 11.58 MB/sec execute 128 sec latency 53.051 ms 2024-09-20T23:48:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: pgmap v20: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-09-20T23:48:24.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:24.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:24 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:25.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:25.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:25.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:25.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: pgmap v20: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-09-20T23:48:25.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:25.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:24 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:25.214 INFO:teuthology.orchestra.run.smithi137.stdout: 5 127209 11.54 MB/sec execute 129 sec latency 59.601 ms 2024-09-20T23:48:26.215 INFO:teuthology.orchestra.run.smithi137.stdout: 5 127531 11.52 MB/sec execute 130 sec latency 80.182 ms 2024-09-20T23:48:27.046 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:26 smithi137 ceph-mon[23698]: pgmap v21: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-09-20T23:48:27.215 INFO:teuthology.orchestra.run.smithi137.stdout: 5 127741 11.47 MB/sec execute 131 sec latency 83.955 ms 2024-09-20T23:48:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:26 smithi144 ceph-mon[26057]: pgmap v21: 97 pgs: 97 active+clean; 346 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.4 MiB/s wr, 428 op/s 2024-09-20T23:48:28.215 INFO:teuthology.orchestra.run.smithi137.stdout: 5 127969 11.43 MB/sec execute 132 sec latency 111.743 ms 2024-09-20T23:48:29.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 128220 11.40 MB/sec execute 133 sec latency 127.817 ms 2024-09-20T23:48:29.355 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:29 smithi137 ceph-mon[23698]: pgmap v22: 97 pgs: 97 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 622 op/s 2024-09-20T23:48:29.355 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:29 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.355 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:29 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.356 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:29 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.356 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:29 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.617 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:29 smithi144 ceph-mon[26057]: pgmap v22: 97 pgs: 97 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.4 MiB/s wr, 622 op/s 2024-09-20T23:48:29.618 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:29 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.618 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:29 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.618 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:29 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:29.618 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:29 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:30.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 128487 11.37 MB/sec execute 134 sec latency 59.658 ms 2024-09-20T23:48:30.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:48:30.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:48:30.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:48:30.399 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:48:30.399 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:48:30.399 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:30 smithi137 ceph-mon[23698]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:48:30.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:48:30.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:48:30.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:48:30.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/278105898' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:48:30.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:48:30.669 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:30 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:48:31.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 128859 11.35 MB/sec execute 135 sec latency 56.551 ms 2024-09-20T23:48:31.414 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:31 smithi137 ceph-mon[23698]: pgmap v23: 97 pgs: 97 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-09-20T23:48:31.414 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:31 smithi137 ceph-mon[23698]: mgrmap e34: smithi137.eurecx(active, since 41s), standbys: smithi144.ygsngz 2024-09-20T23:48:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:31 smithi144 ceph-mon[26057]: pgmap v23: 97 pgs: 97 active+clean; 345 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 430 op/s 2024-09-20T23:48:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:31 smithi144 ceph-mon[26057]: mgrmap e34: smithi137.eurecx(active, since 41s), standbys: smithi144.ygsngz 2024-09-20T23:48:32.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 129163 11.31 MB/sec execute 136 sec latency 64.049 ms 2024-09-20T23:48:33.032 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: pgmap v24: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:33 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: pgmap v24: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 6.0 MiB/s wr, 600 op/s 2024-09-20T23:48:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:33 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:33.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 129433 11.26 MB/sec execute 137 sec latency 87.511 ms 2024-09-20T23:48:34.088 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: Upgrade: Setting container_image for all mgr 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.eurecx"}]: dispatch 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.eurecx"}]': finished 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi144.ygsngz"}]: dispatch 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi144.ygsngz"}]': finished 2024-09-20T23:48:34.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:34 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T23:48:34.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 129794 11.25 MB/sec execute 138 sec latency 54.956 ms 2024-09-20T23:48:34.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: Upgrade: Setting container_image for all mgr 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.eurecx"}]: dispatch 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi137.eurecx"}]': finished 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi144.ygsngz"}]: dispatch 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr.smithi144.ygsngz"}]': finished 2024-09-20T23:48:34.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:34 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T23:48:35.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 130353 11.31 MB/sec execute 139 sec latency 43.472 ms 2024-09-20T23:48:35.359 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:35 smithi137 ceph-mon[23698]: pgmap v25: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-20T23:48:35.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:35 smithi144 ceph-mon[26057]: pgmap v25: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-20T23:48:36.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 130700 11.31 MB/sec execute 140 sec latency 48.149 ms 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: Upgrade: Updating mon.smithi137 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:48:36.380 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:36.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:36 smithi137 ceph-mon[23698]: Deploying daemon mon.smithi137 on smithi137 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: Upgrade: Updating mon.smithi137 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:48:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:36.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:36 smithi144 ceph-mon[26057]: Deploying daemon mon.smithi137 on smithi137 2024-09-20T23:48:37.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 131059 11.31 MB/sec execute 141 sec latency 52.952 ms 2024-09-20T23:48:37.644 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:37 smithi137 ceph-mon[23698]: pgmap v26: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-20T23:48:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:37 smithi144 ceph-mon[26057]: pgmap v26: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.7 MiB/s wr, 363 op/s 2024-09-20T23:48:38.159 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:37 smithi137 systemd[1]: Stopping Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:48:38.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 131320 11.26 MB/sec execute 142 sec latency 90.767 ms 2024-09-20T23:48:38.410 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:38 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[23694]: 2024-09-20T23:48:38.344+0000 7fb187035700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi137 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:48:38.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:38 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[23694]: 2024-09-20T23:48:38.344+0000 7fb187035700 -1 mon.smithi137@0(leader) e2 *** Got Signal Terminated *** 2024-09-20T23:48:38.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:38 smithi137 podman[65059]: 2024-09-20 23:48:38.36550491 +0000 UTC m=+0.242759590 container died fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , version=8, com.redhat.component=centos-stream-container, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, ceph=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_BRANCH=HEAD, io.openshift.expose-services=, vendor=Red Hat, Inc., architecture=x86_64, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-09-20T23:48:39.186 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 podman[65059]: 2024-09-20 23:48:39.072846436 +0000 UTC m=+0.950101110 container cleanup fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, architecture=x86_64, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, version=8) 2024-09-20T23:48:39.186 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 bash[65059]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137 2024-09-20T23:48:39.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 131599 11.22 MB/sec execute 143 sec latency 53.371 ms 2024-09-20T23:48:39.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 podman[65073]: 2024-09-20 23:48:39.544016493 +0000 UTC m=+1.172153037 container remove fe28981f22adef5b1fb320cb0fe4a46ec964582008d6db9ae37dd6e369fa6112 (image=quay.io/ceph/ceph:v17.2.0, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, architecture=x86_64, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, release=754, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, version=8, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-09-20T23:48:39.975 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:48:40.199 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service: Deactivated successfully. 2024-09-20T23:48:40.199 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 systemd[1]: Stopped Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:48:40.199 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:39 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service: Consumed 15.810s CPU time. 2024-09-20T23:48:40.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 131855 11.20 MB/sec execute 144 sec latency 60.306 ms 2024-09-20T23:48:40.454 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:40 smithi137 systemd[1]: Starting Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:48:40.708 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:40 smithi137 podman[65257]: 2024-09-20 23:48:40.44486968 +0000 UTC m=+0.026560062 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:48:40.708 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:40 smithi137 podman[65257]: 2024-09-20 23:48:40.593025232 +0000 UTC m=+0.174715595 container create a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (2m) 8s ago 11m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (10m) 8s ago 10m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (10m) 12s ago 10m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (116s) 8s ago 10m 47.2M - dad864ee21e9 c491347d0b92 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (4m) 8s ago 4m 39.9M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (4m) 12s ago 4m 13.5M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (68s) 8s ago 12m 492M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:48:40.911 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (17s) 12s ago 10m 89.0M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (12m) 8s ago 12m 56.2M 2048M 17.2.0 e1d6a67b021e fe28981f22ad 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (10m) 12s ago 10m 44.0M 2048M 17.2.0 e1d6a67b021e 996acc626286 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (4m) 8s ago 4m 264M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (4m) 12s ago 4m 79.9M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (104s) 8s ago 10m 8170k - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (76s) 12s ago 10m 8674k - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (9m) 8s ago 9m 453M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (8m) 8s ago 8m 405M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (8m) 8s ago 8m 274M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (8m) 8s ago 8m 422M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (7m) 12s ago 7m 441M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (7m) 12s ago 7m 480M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (6m) 12s ago 6m 413M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (6m) 12s ago 6m 572M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:48:40.912 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (24s) 8s ago 9m 44.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 podman[65257]: 2024-09-20 23:48:41.063111518 +0000 UTC m=+0.644801891 container init a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 podman[65257]: 2024-09-20 23:48:41.078193994 +0000 UTC m=+0.659884366 container start a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True) 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: pidfile_write: ignore empty --pid-file 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: load: jerasure load: lrc 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: RocksDB version: 7.9.2 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Git sha 0 2024-09-20T23:48:41.212 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: DB SUMMARY 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: DB Session ID: RT15ZUEK6YT0SM9E7N1L 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: CURRENT file: CURRENT 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: MANIFEST file: MANIFEST-000015 size: 979 Bytes 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi137/store.db dir, Total Num: 1, files: 000030.sst 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi137/store.db: 000028.log size: 1996523 ; 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.error_if_exists: 0 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.create_if_missing: 0 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.paranoid_checks: 1 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T23:48:41.213 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T23:48:41.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.env: 0x564be9c9d940 2024-09-20T23:48:41.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T23:48:41.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.info_log: 0x564beaebb8c0 2024-09-20T23:48:41.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T23:48:41.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.statistics: (nil) 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.use_fsync: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_log_file_size: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_fallocate: 1 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.use_direct_reads: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.db_log_dir: 2024-09-20T23:48:41.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.wal_dir: 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.write_buffer_manager: 0x564beaecadc0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T23:48:41.216 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.unordered_write: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.row_cache: None 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.wal_filter: None 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.two_write_queues: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.wal_compression: 0 2024-09-20T23:48:41.217 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.atomic_flush: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.log_readahead_size: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_background_jobs: 2 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_background_compactions: -1 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_subcompactions: 1 2024-09-20T23:48:41.218 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_open_files: -1 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_background_flushes: -1 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Compression algorithms supported: 2024-09-20T23:48:41.219 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kZSTD supported: 0 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kXpressCompression supported: 0 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kBZip2Compression supported: 0 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kLZ4Compression supported: 1 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kZlibCompression supported: 1 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: kSnappyCompression supported: 1 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000015 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.merge_operator: 2024-09-20T23:48:41.220 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_filter: None 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564beaebb520) 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_type: 0 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_index_type: 0 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_shortening: 1 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: checksum: 4 2024-09-20T23:48:41.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: no_block_cache: 0 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache: 0x564beaeaf1f0 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_options: 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: capacity : 536870912 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_shard_bits : 4 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: strict_capacity_limit : 0 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_cache_compressed: (nil) 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: persistent_cache: (nil) 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size: 4096 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_size_deviation: 10 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_restart_interval: 16 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: index_block_restart_interval: 1 2024-09-20T23:48:41.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_block_size: 4096 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: partition_filters: 0 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: use_delta_encoding: 1 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: filter_policy: bloomfilter 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: whole_key_filtering: 1 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: verify_compression: 0 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: format_version: 5 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_index_compression: 1 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: block_align: 0 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_auto_readahead_size: 262144 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: prepopulate_block_cache: 0 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: initial_auto_readahead_size: 8192 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T23:48:41.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression: NoCompression 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.num_levels: 7 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:48:41.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T23:48:41.225 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T23:48:41.226 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T23:48:41.227 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.inplace_update_support: 0 2024-09-20T23:48:41.228 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.bloom_locality: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.max_successive_merges: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.ttl: 2592000 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T23:48:41.229 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enable_blob_files: false 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.min_blob_size: 0 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 30.sst 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi137/store.db/MANIFEST-000015 succeeded,manifest_file_number is 15, next_file_number is 32, last_sequence is 11873, log_number is 28,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 28 2024-09-20T23:48:41.230 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 959a3a4d-5c4c-4cc0-a40c-4ff068cf2f4d 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876121111800, "job": 1, "event": "recovery_started", "wal_files": [28]} 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #28 mode 2 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876121119166, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 33, "file_size": 1815711, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 11874, "largest_seqno": 12772, "table_properties": {"data_size": 1810925, "index_size": 2722, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1029, "raw_key_size": 9048, "raw_average_key_size": 23, "raw_value_size": 1802378, "raw_average_value_size": 4645, "num_data_blocks": 125, "num_entries": 388, "num_filter_entries": 388, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726876121, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "959a3a4d-5c4c-4cc0-a40c-4ff068cf2f4d", "db_session_id": "RT15ZUEK6YT0SM9E7N1L", "orig_file_number": 33, "seqno_to_time_mapping": "N/A"}} 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876121119404, "job": 1, "event": "recovery_finished"} 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:5047] Creating manifest 35 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi137/store.db/000028.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564beaedce00 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: DB pointer 0x564beafd8000 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** DB Stats ** 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:48:41.231 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L0 1/0 1.73 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 385.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: L6 1/0 7.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Sum 2/0 8.84 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 385.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 385.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T23:48:41.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 385.2 0.00 0.00 1 0.004 0 0 0.0 0.0 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Flush(GB): cumulative 0.002, interval 0.002 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Cumulative compaction: 0.00 GB write, 166.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:48:41.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Interval compaction: 0.00 GB write, 166.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Block cache BinnedLRUCache@0x564beaeaf1f0#2 capacity: 512.00 MB usage: 3.98 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.16 KB,0.000220537%) IndexBlock(1,2.83 KB,0.000539422%) Misc(1,0.00 KB,0%) 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: starting mon.smithi137 rank 0 at public addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] at bind addrs [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi137 fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???) e2 preinit fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).mds e7 new map 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).mds e7 print_map 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: e7 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: legacy client fscid: 1 2024-09-20T23:48:41.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Filesystem 'foofs' (1) 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: fs_name foofs 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: epoch 5 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: created 2024-09-20T23:43:38.248249+0000 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: modified 2024-09-20T23:43:43.412342+0000 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: tableserver 0 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: root 0 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: session_timeout 60 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: session_autoclose 300 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_file_size 1099511627776 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_xattr_size 65536 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: required_client_features {} 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: last_failure 0 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: last_failure_osd_epoch 0 2024-09-20T23:48:41.235 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: max_mds 1 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: in 0 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: up {0=14484} 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: failed 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: damaged 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: stopped 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: data_pools [3] 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: metadata_pool 2 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: inline_data disabled 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: balancer 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: bal_rank_mask -1 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: standby_count_wanted 1 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: qdb_cluster leader: 0 members: 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.foofs.smithi137.qhqtcp{0:14484} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] compat {c=[1],r=[1],i=[7ff]}] 2024-09-20T23:48:41.236 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Standby daemons: 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: [mds.foofs.smithi144.ltxxzn{-1:24295} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] compat {c=[1],r=[1],i=[7ff]}] 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).osd e72 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).osd e72 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:48:41.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137@-1(???).paxosservice(auth 1..29) refresh upgraded, format 0 -> 3 2024-09-20T23:48:41.237 INFO:teuthology.orchestra.run.smithi137.stdout: 5 132123 11.17 MB/sec execute 145 sec latency 85.341 ms 2024-09-20T23:48:41.469 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 bash[65257]: a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d 2024-09-20T23:48:41.469 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 systemd[1]: Started Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:41.493 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 12, 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:48:41.494 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:48:41.977 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:48:41.977 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:48:41.977 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 388 op/s 2024-09-20T23:48:41.977 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='client.24597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/224836059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137 calling monitor election 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: monmap epoch 2 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: last_changed 2024-09-20T23:38:30.082097+0000 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: created 2024-09-20T23:36:10.055803+0000 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: min_mon_release 17 (quincy) 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: election_strategy: 1 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: 0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.smithi137 2024-09-20T23:48:41.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.smithi144 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mgrmap e34: smithi137.eurecx(active, since 52s), standbys: smithi144.ygsngz 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: overall HEALTH_OK 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: from='mgr.14752 ' entity='' 2024-09-20T23:48:41.979 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:41 smithi137 ceph-mon[65321]: mgrmap e35: smithi137.eurecx(active, since 52s), standbys: smithi144.ygsngz 2024-09-20T23:48:42.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='mgr.14752 172.21.15.137:0/1870639311' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: pgmap v28: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.0 MiB/s wr, 388 op/s 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='client.24597 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='client.24601 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='client.24605 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/224836059' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: mon.smithi137 calling monitor election 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: monmap epoch 2 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: last_changed 2024-09-20T23:38:30.082097+0000 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: created 2024-09-20T23:36:10.055803+0000 2024-09-20T23:48:42.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: min_mon_release 17 (quincy) 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: election_strategy: 1 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: 0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.smithi137 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.smithi144 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: osdmap e72: 8 total, 8 up, 8 in 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: mgrmap e34: smithi137.eurecx(active, since 52s), standbys: smithi144.ygsngz 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: overall HEALTH_OK 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: from='mgr.14752 ' entity='' 2024-09-20T23:48:42.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:41 smithi144 ceph-mon[26057]: mgrmap e35: smithi137.eurecx(active, since 52s), standbys: smithi144.ygsngz 2024-09-20T23:48:42.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 132465 11.15 MB/sec execute 146 sec latency 56.847 ms 2024-09-20T23:48:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 132686 11.11 MB/sec execute 147 sec latency 77.001 ms 2024-09-20T23:48:44.089 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:43 smithi137 ceph-mon[65321]: mgrmap e36: smithi137.eurecx(active, since 53s), standbys: smithi144.ygsngz 2024-09-20T23:48:44.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:43 smithi144 ceph-mon[26057]: mgrmap e36: smithi137.eurecx(active, since 53s), standbys: smithi144.ygsngz 2024-09-20T23:48:44.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 132944 11.07 MB/sec execute 148 sec latency 64.282 ms 2024-09-20T23:48:45.216 INFO:teuthology.orchestra.run.smithi137.stdout: 5 133196 11.05 MB/sec execute 149 sec latency 61.693 ms 2024-09-20T23:48:46.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 133587 11.04 MB/sec execute 150 sec latency 64.765 ms 2024-09-20T23:48:47.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 134089 11.07 MB/sec execute 151 sec latency 25.031 ms 2024-09-20T23:48:48.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:48:48.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:48:48.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:48:48.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:48:48.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:48:48.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:47 smithi137 ceph-mon[65321]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:48:48.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz restarted 2024-09-20T23:48:48.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: Standby manager daemon smithi144.ygsngz started 2024-09-20T23:48:48.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/crt"}]: dispatch 2024-09-20T23:48:48.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-20T23:48:48.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi144.ygsngz/key"}]: dispatch 2024-09-20T23:48:48.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:47 smithi144 ceph-mon[26057]: from='mgr.? 172.21.15.144:0/2448307375' entity='mgr.smithi144.ygsngz' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-20T23:48:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 134398 11.09 MB/sec execute 152 sec latency 53.326 ms 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: mgrmap e37: smithi137.eurecx(active, since 58s), standbys: smithi144.ygsngz 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: Active manager daemon smithi137.eurecx restarted 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: Activating manager daemon smithi137.eurecx 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: mgrmap e38: smithi137.eurecx(active, starting, since 0.00676757s), standbys: smithi144.ygsngz 2024-09-20T23:48:49.019 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:48:49.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:48:49.021 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:48:49.145 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:48:49.145 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:48:49.145 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:48:49.145 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "2/23 daemons upgraded", 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "", 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:48:49.146 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:48:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: mgrmap e37: smithi137.eurecx(active, since 58s), standbys: smithi144.ygsngz 2024-09-20T23:48:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: Active manager daemon smithi137.eurecx restarted 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: Activating manager daemon smithi137.eurecx 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: mgrmap e38: smithi137.eurecx(active, starting, since 0.00676757s), standbys: smithi144.ygsngz 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi137.eurecx", "id": "smithi137.eurecx"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr metadata", "who": "smithi144.ygsngz", "id": "smithi144.ygsngz"}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:48:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: Manager daemon smithi137.eurecx is now available 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/mirror_snapshot_schedule"}]: dispatch 2024-09-20T23:48:49.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:48 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi137.eurecx/trash_purge_schedule"}]: dispatch 2024-09-20T23:48:49.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 134632 11.05 MB/sec execute 153 sec latency 40.868 ms 2024-09-20T23:48:49.664 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:48:50.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 134877 11.01 MB/sec execute 154 sec latency 36.293 ms 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: mgrmap e39: smithi137.eurecx(active, since 1.01206s), standbys: smithi144.ygsngz 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: [20/Sep/2024:23:48:49] ENGINE Bus STARTING 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1886179078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: [20/Sep/2024:23:48:49] ENGINE Serving on http://172.21.15.137:8765 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: [20/Sep/2024:23:48:49] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: [20/Sep/2024:23:48:49] ENGINE Bus STARTED 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:50 smithi137 ceph-mon[65321]: [20/Sep/2024:23:48:49] ENGINE Error in HTTPServer.serve 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: Traceback (most recent call last): 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:48:50.290 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._run(expiration_interval) 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: s = self.context.wrap_socket( 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: return self.sslsocket_class._create( 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self.do_handshake() 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: self._sslobj.do_handshake() 2024-09-20T23:48:50.291 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:48:50.405 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: mgrmap e39: smithi137.eurecx(active, since 1.01206s), standbys: smithi144.ygsngz 2024-09-20T23:48:50.405 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: [20/Sep/2024:23:48:49] ENGINE Bus STARTING 2024-09-20T23:48:50.405 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: from='client.? 172.21.15.137:0/1886179078' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:48:50.405 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: [20/Sep/2024:23:48:49] ENGINE Serving on http://172.21.15.137:8765 2024-09-20T23:48:50.405 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: [20/Sep/2024:23:48:49] ENGINE Serving on https://172.21.15.137:7150 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: [20/Sep/2024:23:48:49] ENGINE Bus STARTED 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:50 smithi144 ceph-mon[26057]: [20/Sep/2024:23:48:49] ENGINE Error in HTTPServer.serve 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Traceback (most recent call last): 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._connections.run(self.expiration_interval) 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._run(expiration_interval) 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: s = self.context.wrap_socket( 2024-09-20T23:48:50.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: return self.sslsocket_class._create( 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self.do_handshake() 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: self._sslobj.do_handshake() 2024-09-20T23:48:50.407 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-20T23:48:51.221 INFO:teuthology.orchestra.run.smithi137.stdout: 5 135158 10.97 MB/sec execute 155 sec latency 71.703 ms 2024-09-20T23:48:51.309 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:51 smithi137 ceph-mon[65321]: pgmap v4: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-20T23:48:51.309 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:51 smithi137 ceph-mon[65321]: mgrmap e40: smithi137.eurecx(active, since 2s), standbys: smithi144.ygsngz 2024-09-20T23:48:51.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:51 smithi144 ceph-mon[26057]: pgmap v4: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-20T23:48:51.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:51 smithi144 ceph-mon[26057]: mgrmap e40: smithi137.eurecx(active, since 2s), standbys: smithi144.ygsngz 2024-09-20T23:48:52.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 135578 10.99 MB/sec execute 156 sec latency 35.710 ms 2024-09-20T23:48:53.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 136051 11.00 MB/sec execute 157 sec latency 42.461 ms 2024-09-20T23:48:53.346 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:53 smithi137 ceph-mon[65321]: pgmap v5: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-20T23:48:53.347 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:53 smithi137 ceph-mon[65321]: mgrmap e41: smithi137.eurecx(active, since 4s), standbys: smithi144.ygsngz 2024-09-20T23:48:53.347 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:53 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:53.347 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:53 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:53 smithi144 ceph-mon[26057]: pgmap v5: 97 pgs: 97 active+clean; 343 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail 2024-09-20T23:48:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:53 smithi144 ceph-mon[26057]: mgrmap e41: smithi137.eurecx(active, since 4s), standbys: smithi144.ygsngz 2024-09-20T23:48:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:53 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:53 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:54.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 136341 10.95 MB/sec execute 158 sec latency 47.357 ms 2024-09-20T23:48:54.918 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:54.918 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:54 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:54 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:55.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 136581 10.91 MB/sec execute 159 sec latency 80.803 ms 2024-09-20T23:48:55.850 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:55 smithi144 ceph-mon[26057]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 KiB/s rd, 988 KiB/s wr, 65 op/s 2024-09-20T23:48:55.968 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:55 smithi137 ceph-mon[65321]: pgmap v6: 97 pgs: 97 active+clean; 337 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 49 KiB/s rd, 988 KiB/s wr, 65 op/s 2024-09-20T23:48:56.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 136920 10.90 MB/sec execute 160 sec latency 57.805 ms 2024-09-20T23:48:57.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 137330 10.91 MB/sec execute 161 sec latency 45.877 ms 2024-09-20T23:48:57.768 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 380 op/s 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:57.769 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: pgmap v7: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 35 KiB/s rd, 3.3 MiB/s wr, 380 op/s 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:57.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:48:57.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:57 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:48:58.222 INFO:teuthology.orchestra.run.smithi137.stdout: 5 137595 10.90 MB/sec execute 162 sec latency 57.039 ms 2024-09-20T23:48:58.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:48:58.530 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:48:58.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:58.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:58.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:58.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:58.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:58 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-20T23:48:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.conf 2024-09-20T23:48:58.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:58.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.conf 2024-09-20T23:48:58.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:58.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:58.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:58 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.229 INFO:teuthology.orchestra.run.smithi137.stdout: 5 137968 10.94 MB/sec execute 163 sec latency 56.750 ms 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:59.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:59.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:48:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T23:48:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: Updating smithi144:/etc/ceph/ceph.client.admin.keyring 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: pgmap v8: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 28 KiB/s rd, 2.6 MiB/s wr, 296 op/s 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: Updating smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: Updating smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/config/ceph.client.admin.keyring 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:48:59.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:48:59 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "quorum_status"}]: dispatch 2024-09-20T23:49:00.228 INFO:teuthology.orchestra.run.smithi137.stdout: 5 138303 10.90 MB/sec execute 164 sec latency 42.029 ms 2024-09-20T23:49:01.228 INFO:teuthology.orchestra.run.smithi137.stdout: 5 138554 10.87 MB/sec execute 165 sec latency 86.447 ms 2024-09-20T23:49:01.359 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: pgmap v9: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: Upgrade: Updating mon.smithi144 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:01.360 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:01 smithi137 ceph-mon[65321]: Deploying daemon mon.smithi144 on smithi144 2024-09-20T23:49:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: pgmap v9: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 26 KiB/s rd, 4.4 MiB/s wr, 448 op/s 2024-09-20T23:49:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: Upgrade: Updating mon.smithi144 2024-09-20T23:49:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:01 smithi144 ceph-mon[26057]: Deploying daemon mon.smithi144 on smithi144 2024-09-20T23:49:02.229 INFO:teuthology.orchestra.run.smithi137.stdout: 5 138864 10.86 MB/sec execute 166 sec latency 63.162 ms 2024-09-20T23:49:02.309 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 systemd[1]: Stopping Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:49:02.684 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144[26042]: 2024-09-20T23:49:02.306+0000 7f149f0f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi144 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:49:02.685 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144[26042]: 2024-09-20T23:49:02.306+0000 7f149f0f5700 -1 mon.smithi144@1(peon) e2 *** Got Signal Terminated *** 2024-09-20T23:49:02.685 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 podman[51669]: 2024-09-20 23:49:02.388749154 +0000 UTC m=+0.195203986 container died 996acc6262866f618033b6e548bd59eca27b0836a4c04e525bf7700cb08888cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, distribution-scope=public, release=754, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-09-20T23:49:02.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 podman[51669]: 2024-09-20 23:49:02.824515759 +0000 UTC m=+0.630970581 container cleanup 996acc6262866f618033b6e548bd59eca27b0836a4c04e525bf7700cb08888cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, vcs-type=git, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, name=centos-stream, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., ceph=True, architecture=x86_64, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, io.openshift.expose-services=, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-20T23:49:02.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:02 smithi144 bash[51669]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144 2024-09-20T23:49:03.228 INFO:teuthology.orchestra.run.smithi137.stdout: 5 139197 10.85 MB/sec execute 167 sec latency 52.820 ms 2024-09-20T23:49:03.332 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 podman[51681]: 2024-09-20 23:49:03.077976287 +0000 UTC m=+0.693046322 container remove 996acc6262866f618033b6e548bd59eca27b0836a4c04e525bf7700cb08888cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, CEPH_POINT_RELEASE=-17.2.0, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, vcs-type=git, GIT_CLEAN=True, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, name=centos-stream) 2024-09-20T23:49:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service: Deactivated successfully. 2024-09-20T23:49:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 systemd[1]: Stopped Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:49:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service: Consumed 6.871s CPU time. 2024-09-20T23:49:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 systemd[1]: Starting Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:49:04.066 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 podman[51811]: 2024-09-20 23:49:03.678741289 +0000 UTC m=+0.021980264 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:49:04.066 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:03 smithi144 podman[51811]: 2024-09-20 23:49:03.811709415 +0000 UTC m=+0.154948377 container create 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T23:49:04.066 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 podman[51811]: 2024-09-20 23:49:04.065133071 +0000 UTC m=+0.408372048 container init 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS) 2024-09-20T23:49:04.229 INFO:teuthology.orchestra.run.smithi137.stdout: 5 139814 10.86 MB/sec execute 168 sec latency 57.784 ms 2024-09-20T23:49:04.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:04 smithi137 ceph-mon[65321]: pgmap v10: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-09-20T23:49:04.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:04 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:04.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 podman[51811]: 2024-09-20 23:49:04.070085317 +0000 UTC m=+0.413324289 container start 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:49:04.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: set uid:gid to 167:167 (ceph:ceph) 2024-09-20T23:49:04.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 2 2024-09-20T23:49:04.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: pidfile_write: ignore empty --pid-file 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: load: jerasure load: lrc 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: RocksDB version: 7.9.2 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Git sha 0 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: DB SUMMARY 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: DB Session ID: 45Y4M9ZOO8TUYT5SJK5G 2024-09-20T23:49:04.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: CURRENT file: CURRENT 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: IDENTITY file: IDENTITY 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: MANIFEST file: MANIFEST-000009 size: 904 Bytes 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi144/store.db dir, Total Num: 1, files: 000024.sst 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi144/store.db: 000022.log size: 6225088 ; 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.error_if_exists: 0 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.create_if_missing: 0 2024-09-20T23:49:04.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.paranoid_checks: 1 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.env: 0x55e9a5768940 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.fs: PosixFileSystem 2024-09-20T23:49:04.447 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.info_log: 0x55e9a61798c0 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_file_opening_threads: 16 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.statistics: (nil) 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.use_fsync: 0 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_log_file_size: 0 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-20T23:49:04.448 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.keep_log_file_num: 1000 2024-09-20T23:49:04.449 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.recycle_log_file_num: 0 2024-09-20T23:49:04.449 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_fallocate: 1 2024-09-20T23:49:04.450 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_mmap_reads: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_mmap_writes: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.use_direct_reads: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.create_missing_column_families: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.db_log_dir: 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.wal_dir: 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-20T23:49:04.451 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.advise_random_on_open: 1 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.db_write_buffer_size: 0 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.write_buffer_manager: 0x55e9a6188dc0 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-20T23:49:04.452 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.rate_limiter: (nil) 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.wal_recovery_mode: 2 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enable_thread_tracking: 0 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enable_pipelined_write: 0 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.unordered_write: 0 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-20T23:49:04.453 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.row_cache: None 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.wal_filter: None 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-20T23:49:04.454 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_ingest_behind: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.two_write_queues: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.manual_wal_flush: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.wal_compression: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.atomic_flush: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.log_readahead_size: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.best_efforts_recovery: 0 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-20T23:49:04.455 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.allow_data_in_errors: 0 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.db_host_id: __hostname__ 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_background_jobs: 2 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_background_compactions: -1 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_subcompactions: 1 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_total_wal_size: 0 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-20T23:49:04.456 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_open_files: -1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bytes_per_sync: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_readahead_size: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_background_flushes: -1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Compression algorithms supported: 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kZSTD supported: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kXpressCompression supported: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kBZip2Compression supported: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kLZ4Compression supported: 1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kZlibCompression supported: 1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kLZ4HCCompression supported: 1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: kSnappyCompression supported: 1 2024-09-20T23:49:04.457 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi144/store.db/MANIFEST-000009 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.merge_operator: 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_filter: None 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_filter_factory: None 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.sst_partitioner_factory: None 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55e9a6179520) 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: cache_index_and_filter_blocks: 1 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-20T23:49:04.458 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: pin_top_level_index_and_filter: 1 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: index_type: 0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: data_block_index_type: 0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: index_shortening: 1 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: checksum: 4 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: no_block_cache: 0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_cache: 0x55e9a616d1f0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_cache_name: BinnedLRUCache 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_cache_options: 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: capacity : 536870912 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: num_shard_bits : 4 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: strict_capacity_limit : 0 2024-09-20T23:49:04.459 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: high_pri_pool_ratio: 0.000 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_cache_compressed: (nil) 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: persistent_cache: (nil) 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_size: 4096 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_size_deviation: 10 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_restart_interval: 16 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: index_block_restart_interval: 1 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: metadata_block_size: 4096 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: partition_filters: 0 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: use_delta_encoding: 1 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: filter_policy: bloomfilter 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: whole_key_filtering: 1 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: verify_compression: 0 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: read_amp_bytes_per_bit: 0 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: format_version: 5 2024-09-20T23:49:04.460 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: enable_index_compression: 1 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: block_align: 0 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: max_auto_readahead_size: 262144 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: prepopulate_block_cache: 0 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: initial_auto_readahead_size: 8192 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: num_file_reads_for_auto_readahead: 2 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.write_buffer_size: 33554432 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_write_buffer_number: 2 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression: NoCompression 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression: Disabled 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.prefix_extractor: nullptr 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.num_levels: 7 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-20T23:49:04.461 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.level: 32767 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.strategy: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-20T23:49:04.462 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.enabled: false 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.target_file_size_base: 67108864 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-20T23:49:04.463 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.arena_block_size: 1048576 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.disable_auto_compactions: 0 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-20T23:49:04.464 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.inplace_update_support: 0 2024-09-20T23:49:04.465 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.bloom_locality: 0 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.max_successive_merges: 0 2024-09-20T23:49:04.466 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.paranoid_file_checks: 0 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.force_consistency_checks: 1 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.report_bg_io_stats: 0 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.ttl: 2592000 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-20T23:49:04.467 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enable_blob_files: false 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.min_blob_size: 0 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_file_size: 268435456 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-20T23:49:04.468 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.blob_file_starting_level: 0 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [table/block_based/block_based_table_reader.cc:721] At least one SST file opened without unique ID to verify: 24.sst 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi144/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 26, last_sequence is 12239, log_number is 22,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 22 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 268ff3ed-51c7-459d-8a29-70ffaeb9f33a 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876144102772, "job": 1, "event": "recovery_started", "wal_files": [22]} 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #22 mode 2 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876144118665, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 27, "file_size": 3761313, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 12244, "largest_seqno": 13741, "table_properties": {"data_size": 3754408, "index_size": 4324, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 1541, "raw_key_size": 14097, "raw_average_key_size": 23, "raw_value_size": 3741197, "raw_average_value_size": 6194, "num_data_blocks": 197, "num_entries": 604, "num_filter_entries": 604, "num_deletions": 2, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726876144, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "268ff3ed-51c7-459d-8a29-70ffaeb9f33a", "db_session_id": "45Y4M9ZOO8TUYT5SJK5G", "orig_file_number": 27, "seqno_to_time_mapping": "N/A"}} 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726876144118839, "job": 1, "event": "recovery_finished"} 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:5047] Creating manifest 29 2024-09-20T23:49:04.469 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/version_set.cc:4390] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed. 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-smithi144/store.db/000022.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55e9a619ae00 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: DB pointer 0x55e9a6296000 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ** DB Stats ** 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-20T23:49:04.470 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ** Compaction Stats [default] ** 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: L0 1/0 3.59 MB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 533.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: L6 1/0 7.11 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 0.0 0.0 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Sum 2/0 10.70 MB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 533.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 533.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ** Compaction Stats [default] ** 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 533.6 0.01 0.00 1 0.007 0 0 0.0 0.0 2024-09-20T23:49:04.471 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Flush(GB): cumulative 0.004, interval 0.004 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Cumulative compaction: 0.00 GB write, 139.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Interval compaction: 0.00 GB write, 139.35 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Block cache BinnedLRUCache@0x55e9a616d1f0#2 capacity: 512.00 MB usage: 6.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.4e-05 secs_since: 0 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,1.78 KB,0.000339746%) IndexBlock(1,4.58 KB,0.000873208%) Misc(1,0.00 KB,0%) 2024-09-20T23:49:04.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: starting mon.smithi144 rank 1 at public addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] at bind addrs [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi144 fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???) e2 preinit fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).mds e7 new map 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).mds e7 print_map 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: e7 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: btime 1970-01-01T00:00:00:000000+0000 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: legacy client fscid: 1 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Filesystem 'foofs' (1) 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: fs_name foofs 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: epoch 5 2024-09-20T23:49:04.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: flags 12 joinable allow_snaps allow_multimds_snaps 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: created 2024-09-20T23:43:38.248249+0000 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: modified 2024-09-20T23:43:43.412342+0000 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: tableserver 0 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: root 0 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: session_timeout 60 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: session_autoclose 300 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: max_file_size 1099511627776 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: max_xattr_size 65536 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: required_client_features {} 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: last_failure 0 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: last_failure_osd_epoch 0 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: max_mds 1 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: in 0 2024-09-20T23:49:04.474 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: up {0=14484} 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: failed 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: damaged 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: stopped 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: data_pools [3] 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: metadata_pool 2 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: inline_data disabled 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: balancer 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: bal_rank_mask -1 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: standby_count_wanted 1 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: qdb_cluster leader: 0 members: 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: [mds.foofs.smithi137.qhqtcp{0:14484} state up:active seq 2 join_fscid=1 addr [v2:172.21.15.137:6834/408306656,v1:172.21.15.137:6835/408306656] compat {c=[1],r=[1],i=[7ff]}] 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.475 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: Standby daemons: 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout: [mds.foofs.smithi144.ltxxzn{-1:24295} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] compat {c=[1],r=[1],i=[7ff]}] 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).osd e73 crush map has features 3314933000854323200, adjusting msgr requires 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).osd e73 crush map has features 432629239337189376, adjusting msgr requires 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: mon.smithi144@-1(???).paxosservice(auth 1..30) refresh upgraded, format 0 -> 3 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 bash[51811]: 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 systemd[1]: Started Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: pgmap v10: 97 pgs: 97 active+clean; 346 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 23 KiB/s rd, 4.0 MiB/s wr, 410 op/s 2024-09-20T23:49:04.476 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:04 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:05.229 INFO:teuthology.orchestra.run.smithi137.stdout: 5 140614 10.91 MB/sec execute 169 sec latency 21.367 ms 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: pgmap v11: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: mon.smithi137 calling monitor election 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: mon.smithi144 calling monitor election 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: monmap epoch 3 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: last_changed 2024-09-20T23:49:04.233597+0000 2024-09-20T23:49:05.416 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: created 2024-09-20T23:36:10.055803+0000 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: min_mon_release 19 (squid) 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: election_strategy: 1 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: 0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.smithi137 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.smithi144 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: mgrmap e41: smithi137.eurecx(active, since 16s), standbys: smithi144.ygsngz 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: overall HEALTH_OK 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:05.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:49:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: pgmap v11: 97 pgs: 97 active+clean; 342 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 27 KiB/s rd, 4.4 MiB/s wr, 426 op/s 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi137"}]: dispatch 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: mon.smithi137 calling monitor election 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: mon.smithi144 calling monitor election 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mon metadata", "id": "smithi144"}]: dispatch 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: mon.smithi137 is new leader, mons smithi137,smithi144 in quorum (ranks 0,1) 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: monmap epoch 3 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: last_changed 2024-09-20T23:49:04.233597+0000 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: created 2024-09-20T23:36:10.055803+0000 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: min_mon_release 19 (squid) 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: election_strategy: 1 2024-09-20T23:49:05.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: 0: [v2:172.21.15.137:3300/0,v1:172.21.15.137:6789/0] mon.smithi137 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: 1: [v2:172.21.15.144:3300/0,v1:172.21.15.144:6789/0] mon.smithi144 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: osdmap e73: 8 total, 8 up, 8 in 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: mgrmap e41: smithi137.eurecx(active, since 16s), standbys: smithi144.ygsngz 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: overall HEALTH_OK 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:05.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:49:06.229 INFO:teuthology.orchestra.run.smithi137.stdout: 5 141076 10.94 MB/sec execute 170 sec latency 57.891 ms 2024-09-20T23:49:07.235 INFO:teuthology.orchestra.run.smithi137.stdout: 5 141707 11.00 MB/sec execute 171 sec latency 30.558 ms 2024-09-20T23:49:07.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:07 smithi137 ceph-mon[65321]: pgmap v12: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.8 MiB/s wr, 614 op/s 2024-09-20T23:49:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:07 smithi144 ceph-mon[51829]: pgmap v12: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.8 MiB/s wr, 614 op/s 2024-09-20T23:49:08.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 142032 10.98 MB/sec execute 172 sec latency 43.284 ms 2024-09-20T23:49:09.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 142273 10.94 MB/sec execute 173 sec latency 66.581 ms 2024-09-20T23:49:09.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:09 smithi137 ceph-mon[65321]: pgmap v13: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-09-20T23:49:09.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:09 smithi144 ceph-mon[51829]: pgmap v13: 97 pgs: 97 active+clean; 343 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.3 MiB/s wr, 419 op/s 2024-09-20T23:49:09.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:09.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:10.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 142728 10.96 MB/sec execute 174 sec latency 52.158 ms 2024-09-20T23:49:11.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 143467 10.99 MB/sec execute 175 sec latency 38.345 ms 2024-09-20T23:49:11.318 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:11 smithi137 ceph-mon[65321]: pgmap v14: 97 pgs: 97 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 713 op/s 2024-09-20T23:49:11.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:11 smithi144 ceph-mon[51829]: pgmap v14: 97 pgs: 97 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.5 MiB/s wr, 713 op/s 2024-09-20T23:49:12.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 143840 10.98 MB/sec execute 176 sec latency 76.964 ms 2024-09-20T23:49:13.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 144424 11.02 MB/sec execute 177 sec latency 42.861 ms 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: pgmap v15: 97 pgs: 97 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 524 op/s 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:13.851 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:13.852 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:13.852 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:13.852 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:14.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: pgmap v15: 97 pgs: 97 active+clean; 349 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 524 op/s 2024-09-20T23:49:14.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:14.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:14.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:14.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:49:14.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:14.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:14.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:14.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:14.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:14.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 144923 11.06 MB/sec execute 178 sec latency 29.864 ms 2024-09-20T23:49:14.858 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:14 smithi137 ceph-mon[65321]: Reconfiguring mon.smithi137 (monmap changed)... 2024-09-20T23:49:14.858 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:14 smithi137 ceph-mon[65321]: Reconfiguring daemon mon.smithi137 on smithi137 2024-09-20T23:49:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:14 smithi144 ceph-mon[51829]: Reconfiguring mon.smithi137 (monmap changed)... 2024-09-20T23:49:15.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:14 smithi144 ceph-mon[51829]: Reconfiguring daemon mon.smithi137 on smithi137 2024-09-20T23:49:15.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 145420 11.06 MB/sec execute 179 sec latency 38.877 ms 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 571 op/s 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:49:15.902 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:16.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: pgmap v16: 97 pgs: 97 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 571 op/s 2024-09-20T23:49:16.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:16.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:16.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi137.eurecx", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:49:16.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:49:16.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:16.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 145746 11.04 MB/sec execute 180 sec latency 57.751 ms 2024-09-20T23:49:16.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:16 smithi137 ceph-mon[65321]: Reconfiguring mgr.smithi137.eurecx (monmap changed)... 2024-09-20T23:49:16.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:16 smithi137 ceph-mon[65321]: Reconfiguring daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:49:17.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:16 smithi144 ceph-mon[51829]: Reconfiguring mgr.smithi137.eurecx (monmap changed)... 2024-09-20T23:49:17.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:16 smithi144 ceph-mon[51829]: Reconfiguring daemon mgr.smithi137.eurecx on smithi137 2024-09-20T23:49:17.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 145976 11.02 MB/sec execute 181 sec latency 77.181 ms 2024-09-20T23:49:17.950 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:17 smithi137 ceph-mon[65321]: pgmap v17: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 8.2 MiB/s wr, 820 op/s 2024-09-20T23:49:17.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:17.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:17.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:17.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:17 smithi144 ceph-mon[51829]: pgmap v17: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 8.2 MiB/s wr, 820 op/s 2024-09-20T23:49:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:18.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 146350 11.00 MB/sec execute 182 sec latency 50.676 ms 2024-09-20T23:49:18.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:18 smithi137 ceph-mon[65321]: Reconfiguring crash.smithi137 (monmap changed)... 2024-09-20T23:49:18.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:18 smithi137 ceph-mon[65321]: Reconfiguring daemon crash.smithi137 on smithi137 2024-09-20T23:49:18.969 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:18 smithi144 ceph-mon[51829]: Reconfiguring crash.smithi137 (monmap changed)... 2024-09-20T23:49:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:18 smithi144 ceph-mon[51829]: Reconfiguring daemon crash.smithi137 on smithi137 2024-09-20T23:49:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:19.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 146759 11.01 MB/sec execute 183 sec latency 58.826 ms 2024-09-20T23:49:19.982 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:19 smithi137 ceph-mon[65321]: pgmap v18: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 605 op/s 2024-09-20T23:49:20.156 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:49:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:19 smithi144 ceph-mon[51829]: pgmap v18: 97 pgs: 97 active+clean; 346 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.4 MiB/s wr, 605 op/s 2024-09-20T23:49:20.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 147048 10.97 MB/sec execute 184 sec latency 67.985 ms 2024-09-20T23:49:21.072 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (2m) 24s ago 11m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (11m) 24s ago 11m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (11m) 12s ago 11m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (2m) 24s ago 11m 54.5M - dad864ee21e9 c491347d0b92 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (5m) 24s ago 5m 39.9M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (5m) 12s ago 5m 13.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (108s) 24s ago 12m 473M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (57s) 12s ago 10m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (40s) 24s ago 13m 28.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (17s) 12s ago 10m 24.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (5m) 24s ago 5m 262M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (5m) 12s ago 5m 81.5M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (2m) 24s ago 11m 8753k - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:49:21.073 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (116s) 12s ago 10m 17.0M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (10m) 24s ago 10m 480M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (9m) 24s ago 9m 408M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (9m) 24s ago 9m 276M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (8m) 24s ago 8m 446M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (8m) 12s ago 8m 486M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (7m) 12s ago 7m 540M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (7m) 12s ago 7m 455M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (7m) 12s ago 7m 356M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:49:21.074 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (64s) 24s ago 10m 49.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:49:21.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 147288 10.94 MB/sec execute 185 sec latency 59.380 ms 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:21.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.0 on smithi137 2024-09-20T23:49:21.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:21.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 837 op/s 2024-09-20T23:49:21.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:21 smithi137 ceph-mon[65321]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: Reconfiguring osd.0 (monmap changed)... 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.0 on smithi137 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='client.34134 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: pgmap v19: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 12 KiB/s rd, 8.6 MiB/s wr, 837 op/s 2024-09-20T23:49:21.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:21 smithi144 ceph-mon[51829]: from='client.34138 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:49:21.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:49:21.631 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:49:22.164 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:49:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "4/23 daemons upgraded", 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading mon daemons", 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:49:22.165 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:49:22.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 147689 10.93 MB/sec execute 186 sec latency 39.776 ms 2024-09-20T23:49:22.281 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:22 smithi137 ceph-mon[65321]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:22.282 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:22 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1673271104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:22 smithi144 ceph-mon[51829]: from='client.44107 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:22 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/1673271104' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:22.853 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:49:23.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 148099 10.96 MB/sec execute 187 sec latency 57.235 ms 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.1 on smithi137 2024-09-20T23:49:23.805 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:23 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2876365287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:49:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: pgmap v20: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.4 MiB/s wr, 542 op/s 2024-09-20T23:49:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='client.44111 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: Reconfiguring osd.1 (monmap changed)... 2024-09-20T23:49:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:49:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.1 on smithi137 2024-09-20T23:49:23.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:23 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2876365287' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:49:24.238 INFO:teuthology.orchestra.run.smithi137.stdout: 5 148511 10.99 MB/sec execute 188 sec latency 45.846 ms 2024-09-20T23:49:25.236 INFO:teuthology.orchestra.run.smithi137.stdout: 5 149004 10.99 MB/sec execute 189 sec latency 25.883 ms 2024-09-20T23:49:25.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:25 smithi137 ceph-mon[65321]: pgmap v21: 97 pgs: 97 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 629 op/s 2024-09-20T23:49:25.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:25.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:25.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:49:25.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:25 smithi144 ceph-mon[51829]: pgmap v21: 97 pgs: 97 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.3 MiB/s wr, 629 op/s 2024-09-20T23:49:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:49:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:26.237 INFO:teuthology.orchestra.run.smithi137.stdout: 5 149431 10.98 MB/sec execute 190 sec latency 30.788 ms 2024-09-20T23:49:26.845 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:26 smithi137 ceph-mon[65321]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T23:49:26.845 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:26 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.2 on smithi137 2024-09-20T23:49:26.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:26 smithi144 ceph-mon[51829]: Reconfiguring osd.2 (monmap changed)... 2024-09-20T23:49:26.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:26 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.2 on smithi137 2024-09-20T23:49:27.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 149702 10.96 MB/sec execute 191 sec latency 45.607 ms 2024-09-20T23:49:27.865 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:27 smithi137 ceph-mon[65321]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.5 MiB/s wr, 704 op/s 2024-09-20T23:49:27.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:27 smithi144 ceph-mon[51829]: pgmap v22: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 6.5 MiB/s wr, 704 op/s 2024-09-20T23:49:28.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 150052 10.98 MB/sec execute 192 sec latency 72.532 ms 2024-09-20T23:49:28.906 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:28.907 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:28.907 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T23:49:28.907 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:49:28.907 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:28.907 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:28 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.3 on smithi137 2024-09-20T23:49:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: Reconfiguring osd.3 (monmap changed)... 2024-09-20T23:49:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:49:28.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:28.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:28 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.3 on smithi137 2024-09-20T23:49:29.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 150397 10.93 MB/sec execute 193 sec latency 45.935 ms 2024-09-20T23:49:29.911 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:29 smithi137 ceph-mon[65321]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-09-20T23:49:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:29 smithi144 ceph-mon[51829]: pgmap v23: 97 pgs: 97 active+clean; 340 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 439 op/s 2024-09-20T23:49:30.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 150714 10.90 MB/sec execute 194 sec latency 53.955 ms 2024-09-20T23:49:31.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 150997 10.90 MB/sec execute 195 sec latency 51.251 ms 2024-09-20T23:49:31.693 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 672 op/s 2024-09-20T23:49:31.693 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:31.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:31.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: Reconfiguring mds.foofs.smithi137.qhqtcp (monmap changed)... 2024-09-20T23:49:31.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:49:31.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:31.694 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:31 smithi137 ceph-mon[65321]: Reconfiguring daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:49:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: pgmap v24: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 672 op/s 2024-09-20T23:49:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: Reconfiguring mds.foofs.smithi137.qhqtcp (monmap changed)... 2024-09-20T23:49:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:49:31.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:31.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:31 smithi144 ceph-mon[51829]: Reconfiguring daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:49:32.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 151379 10.90 MB/sec execute 196 sec latency 45.771 ms 2024-09-20T23:49:33.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 151679 10.90 MB/sec execute 197 sec latency 49.917 ms 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: Reconfiguring crash.smithi144 (monmap changed)... 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: Reconfiguring daemon crash.smithi144 on smithi144 2024-09-20T23:49:33.911 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:34.009 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: pgmap v25: 97 pgs: 97 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 440 op/s 2024-09-20T23:49:34.009 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: Reconfiguring crash.smithi144 (monmap changed)... 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: Reconfiguring daemon crash.smithi144 on smithi144 2024-09-20T23:49:34.010 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:34.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 151952 10.91 MB/sec execute 198 sec latency 45.696 ms 2024-09-20T23:49:35.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 152235 10.89 MB/sec execute 199 sec latency 60.996 ms 2024-09-20T23:49:35.784 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: Reconfiguring mgr.smithi144.ygsngz (monmap changed)... 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:35 smithi137 ceph-mon[65321]: Reconfiguring daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:49:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: pgmap v26: 97 pgs: 97 active+clean; 342 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 9.3 KiB/s rd, 5.7 MiB/s wr, 591 op/s 2024-09-20T23:49:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: Reconfiguring mgr.smithi144.ygsngz (monmap changed)... 2024-09-20T23:49:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi144.ygsngz", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-20T23:49:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-20T23:49:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:35 smithi144 ceph-mon[51829]: Reconfiguring daemon mgr.smithi144.ygsngz on smithi144 2024-09-20T23:49:36.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 152663 10.88 MB/sec execute 200 sec latency 43.678 ms 2024-09-20T23:49:37.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 153249 10.89 MB/sec execute 201 sec latency 25.727 ms 2024-09-20T23:49:37.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: pgmap v27: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: Reconfiguring mon.smithi144 (monmap changed)... 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:37.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:37 smithi137 ceph-mon[65321]: Reconfiguring daemon mon.smithi144 on smithi144 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: pgmap v27: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 5.6 MiB/s wr, 541 op/s 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: Reconfiguring mon.smithi144 (monmap changed)... 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-20T23:49:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:37.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:37 smithi144 ceph-mon[51829]: Reconfiguring daemon mon.smithi144 on smithi144 2024-09-20T23:49:38.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 153783 10.91 MB/sec execute 202 sec latency 34.568 ms 2024-09-20T23:49:39.072 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:39.072 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:39.073 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T23:49:39.073 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:49:39.073 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:39.073 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.4 on smithi144 2024-09-20T23:49:39.073 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:39 smithi137 ceph-mon[65321]: pgmap v28: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-09-20T23:49:39.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 154261 10.89 MB/sec execute 203 sec latency 34.904 ms 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: Reconfiguring osd.4 (monmap changed)... 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.4 on smithi144 2024-09-20T23:49:39.381 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:39 smithi144 ceph-mon[51829]: pgmap v28: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.6 KiB/s rd, 4.3 MiB/s wr, 420 op/s 2024-09-20T23:49:40.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 154697 10.91 MB/sec execute 204 sec latency 34.954 ms 2024-09-20T23:49:41.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 155182 10.92 MB/sec execute 205 sec latency 34.079 ms 2024-09-20T23:49:41.629 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: pgmap v29: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 699 op/s 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:41.630 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:41 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.5 on smithi144 2024-09-20T23:49:41.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: pgmap v29: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 7.0 MiB/s wr, 699 op/s 2024-09-20T23:49:41.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:41.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:41.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: Reconfiguring osd.5 (monmap changed)... 2024-09-20T23:49:41.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:49:41.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:41.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:41 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.5 on smithi144 2024-09-20T23:49:42.241 INFO:teuthology.orchestra.run.smithi137.stdout: 5 155792 10.99 MB/sec execute 206 sec latency 51.481 ms 2024-09-20T23:49:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 156411 10.99 MB/sec execute 207 sec latency 37.602 ms 2024-09-20T23:49:43.394 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: pgmap v30: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:43.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:43 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.6 on smithi144 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: pgmap v30: 97 pgs: 97 active+clean; 339 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.8 MiB/s wr, 466 op/s 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: Reconfiguring osd.6 (monmap changed)... 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:49:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:43 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.6 on smithi144 2024-09-20T23:49:44.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 156656 10.97 MB/sec execute 208 sec latency 76.136 ms 2024-09-20T23:49:45.032 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:45.032 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:45.032 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T23:49:45.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:49:45.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:45.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: Reconfiguring daemon osd.7 on smithi144 2024-09-20T23:49:45.033 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:44 smithi137 ceph-mon[65321]: pgmap v31: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 710 op/s 2024-09-20T23:49:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 156925 10.95 MB/sec execute 209 sec latency 68.117 ms 2024-09-20T23:49:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: Reconfiguring osd.7 (monmap changed)... 2024-09-20T23:49:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:49:45.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:45.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: Reconfiguring daemon osd.7 on smithi144 2024-09-20T23:49:45.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:44 smithi144 ceph-mon[51829]: pgmap v31: 97 pgs: 97 active+clean; 346 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.8 MiB/s wr, 710 op/s 2024-09-20T23:49:46.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 157355 10.95 MB/sec execute 210 sec latency 43.725 ms 2024-09-20T23:49:46.809 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:46.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:46.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: Reconfiguring mds.foofs.smithi144.ltxxzn (monmap changed)... 2024-09-20T23:49:46.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:49:46.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:46.810 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:46 smithi137 ceph-mon[65321]: Reconfiguring daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:49:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: Reconfiguring mds.foofs.smithi144.ltxxzn (monmap changed)... 2024-09-20T23:49:46.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:49:46.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:46.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:46 smithi144 ceph-mon[51829]: Reconfiguring daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:49:47.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 157938 10.95 MB/sec execute 211 sec latency 31.406 ms 2024-09-20T23:49:47.816 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.7 MiB/s wr, 610 op/s 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]: dispatch 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]': finished 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi144"}]: dispatch 2024-09-20T23:49:47.817 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi144"}]': finished 2024-09-20T23:49:47.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: pgmap v32: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.3 KiB/s rd, 6.7 MiB/s wr, 610 op/s 2024-09-20T23:49:47.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:49:47.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]: dispatch 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi137"}]': finished 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi144"}]: dispatch 2024-09-20T23:49:47.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon.smithi144"}]': finished 2024-09-20T23:49:48.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 158359 10.96 MB/sec execute 212 sec latency 42.863 ms 2024-09-20T23:49:48.822 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:48 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all mon 2024-09-20T23:49:48.823 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:48 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all mon 2024-09-20T23:49:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:49:49.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 158740 10.98 MB/sec execute 213 sec latency 54.162 ms 2024-09-20T23:49:49.760 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:49 smithi137 ceph-mon[65321]: pgmap v33: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-09-20T23:49:49.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:49 smithi144 ceph-mon[51829]: pgmap v33: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.9 MiB/s wr, 573 op/s 2024-09-20T23:49:50.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 159213 11.01 MB/sec execute 214 sec latency 46.262 ms 2024-09-20T23:49:50.870 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:50 smithi137 ceph-mon[65321]: Upgrade: Updating crash.smithi137 (1/2) 2024-09-20T23:49:50.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:50.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:50.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:50.871 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:50 smithi137 ceph-mon[65321]: Deploying daemon crash.smithi137 on smithi137 2024-09-20T23:49:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:50 smithi144 ceph-mon[51829]: Upgrade: Updating crash.smithi137 (1/2) 2024-09-20T23:49:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi137", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:50 smithi144 ceph-mon[51829]: Deploying daemon crash.smithi137 on smithi137 2024-09-20T23:49:51.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 159567 10.99 MB/sec execute 215 sec latency 35.761 ms 2024-09-20T23:49:51.885 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:51 smithi137 ceph-mon[65321]: pgmap v34: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 823 op/s 2024-09-20T23:49:52.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:51 smithi144 ceph-mon[51829]: pgmap v34: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 8.4 MiB/s wr, 823 op/s 2024-09-20T23:49:52.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 159872 10.97 MB/sec execute 216 sec latency 70.522 ms 2024-09-20T23:49:53.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 160191 10.95 MB/sec execute 217 sec latency 42.500 ms 2024-09-20T23:49:53.488 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:49:53.911 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:53 smithi137 ceph-mon[65321]: pgmap v35: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-09-20T23:49:54.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:53 smithi144 ceph-mon[51829]: pgmap v35: 97 pgs: 97 active+clean; 342 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.9 KiB/s rd, 5.7 MiB/s wr, 544 op/s 2024-09-20T23:49:54.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 160480 10.94 MB/sec execute 218 sec latency 63.386 ms 2024-09-20T23:49:54.602 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (3m) 58s ago 12m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (12m) 58s ago 12m 6409k - 17.2.0 e1d6a67b021e a00b47ac6f2e 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (11m) 46s ago 11m 6400k - 17.2.0 e1d6a67b021e f74e829374f3 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (3m) 58s ago 11m 54.5M - dad864ee21e9 c491347d0b92 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (6m) 58s ago 6m 39.9M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (6m) 46s ago 6m 13.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (2m) 58s ago 13m 473M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (91s) 46s ago 11m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (74s) 58s ago 13m 28.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (50s) 46s ago 11m 24.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (5m) 58s ago 5m 262M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (5m) 46s ago 5m 81.5M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:49:54.603 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (2m) 58s ago 11m 8753k - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (2m) 46s ago 11m 17.0M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (10m) 58s ago 10m 480M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (10m) 58s ago 10m 408M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (9m) 58s ago 9m 276M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (9m) 58s ago 9m 446M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (8m) 46s ago 8m 486M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (8m) 46s ago 8m 540M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (8m) 46s ago 8m 455M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (7m) 46s ago 7m 356M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:49:54.604 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (97s) 58s ago 11m 49.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:49:54.923 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:54 smithi137 ceph-mon[65321]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:55.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:54 smithi144 ceph-mon[51829]: from='client.34158 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:55.215 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:49:55.215 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:49:55.215 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:49:55.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 10, 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:49:55.216 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:49:55.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 160828 10.92 MB/sec execute 219 sec latency 48.919 ms 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "4/23 daemons upgraded", 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading crash daemons", 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:49:55.842 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:49:55.950 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: from='client.44121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:55.950 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 585 op/s 2024-09-20T23:49:55.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:55.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2519197654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:55.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:55.951 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:55 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: from='client.44121 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: pgmap v36: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.6 MiB/s wr, 585 op/s 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: from='client.34166 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2519197654' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:56.065 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:55 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:56.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 161186 10.91 MB/sec execute 220 sec latency 44.493 ms 2024-09-20T23:49:56.453 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:49:56.960 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:56 smithi137 ceph-mon[65321]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:56.960 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:56 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/955891286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:49:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:56 smithi144 ceph-mon[51829]: from='client.34174 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:49:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:56 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/955891286' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:49:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 161566 10.91 MB/sec execute 221 sec latency 32.557 ms 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: pgmap v37: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 522 op/s 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: Upgrade: Updating crash.smithi144 (2/2) 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:57.898 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:57 smithi137 ceph-mon[65321]: Deploying daemon crash.smithi144 on smithi144 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: pgmap v37: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.2 KiB/s rd, 6.0 MiB/s wr, 522 op/s 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: Upgrade: Updating crash.smithi144 (2/2) 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi144", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:49:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:57 smithi144 ceph-mon[51829]: Deploying daemon crash.smithi144 on smithi144 2024-09-20T23:49:58.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 161984 10.90 MB/sec execute 222 sec latency 35.401 ms 2024-09-20T23:49:59.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 162322 10.91 MB/sec execute 223 sec latency 44.855 ms 2024-09-20T23:50:00.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:49:59 smithi137 ceph-mon[65321]: pgmap v38: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-09-20T23:50:00.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:49:59 smithi144 ceph-mon[51829]: pgmap v38: 97 pgs: 97 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.8 MiB/s wr, 471 op/s 2024-09-20T23:50:00.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 162681 10.93 MB/sec execute 224 sec latency 42.183 ms 2024-09-20T23:50:01.053 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:00 smithi137 ceph-mon[65321]: overall HEALTH_OK 2024-09-20T23:50:01.053 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:00 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:01.053 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:00 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:01.053 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:00 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:00 smithi144 ceph-mon[51829]: overall HEALTH_OK 2024-09-20T23:50:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:00 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:00 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:01.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:00 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:01.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 162971 10.91 MB/sec execute 225 sec latency 61.318 ms 2024-09-20T23:50:02.063 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:01 smithi137 ceph-mon[65321]: pgmap v39: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 689 op/s 2024-09-20T23:50:02.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:01 smithi144 ceph-mon[51829]: pgmap v39: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.9 MiB/s wr, 689 op/s 2024-09-20T23:50:02.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 163319 10.90 MB/sec execute 226 sec latency 44.087 ms 2024-09-20T23:50:03.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 163683 10.89 MB/sec execute 227 sec latency 40.078 ms 2024-09-20T23:50:04.133 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:03 smithi137 ceph-mon[65321]: pgmap v40: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-09-20T23:50:04.134 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:03 smithi144 ceph-mon[51829]: pgmap v40: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.5 MiB/s wr, 440 op/s 2024-09-20T23:50:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:04.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 164006 10.88 MB/sec execute 228 sec latency 69.404 ms 2024-09-20T23:50:05.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 164406 10.88 MB/sec execute 229 sec latency 41.914 ms 2024-09-20T23:50:05.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:05 smithi137 ceph-mon[65321]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 619 op/s 2024-09-20T23:50:05.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:05.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:05.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:05.942 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:05 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:06.148 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:05 smithi144 ceph-mon[51829]: pgmap v41: 97 pgs: 97 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.0 MiB/s wr, 619 op/s 2024-09-20T23:50:06.148 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:06.148 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:06.148 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:06.148 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:05 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:06.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 164793 10.85 MB/sec execute 230 sec latency 67.596 ms 2024-09-20T23:50:07.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 165264 10.86 MB/sec execute 231 sec latency 49.249 ms 2024-09-20T23:50:07.975 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:07 smithi137 ceph-mon[65321]: pgmap v42: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 607 op/s 2024-09-20T23:50:08.181 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:07 smithi144 ceph-mon[51829]: pgmap v42: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 5.7 MiB/s wr, 607 op/s 2024-09-20T23:50:08.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 165686 10.87 MB/sec execute 232 sec latency 44.738 ms 2024-09-20T23:50:09.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 166027 10.87 MB/sec execute 233 sec latency 47.308 ms 2024-09-20T23:50:10.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:09 smithi137 ceph-mon[65321]: pgmap v43: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-09-20T23:50:10.008 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:10.008 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:09 smithi144 ceph-mon[51829]: pgmap v43: 97 pgs: 97 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.3 MiB/s wr, 426 op/s 2024-09-20T23:50:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:10.242 INFO:teuthology.orchestra.run.smithi137.stdout: 5 166497 10.90 MB/sec execute 234 sec latency 54.693 ms 2024-09-20T23:50:11.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 166848 10.89 MB/sec execute 235 sec latency 63.808 ms 2024-09-20T23:50:11.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:11 smithi137 ceph-mon[65321]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.7 MiB/s wr, 673 op/s 2024-09-20T23:50:11.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:11 smithi144 ceph-mon[51829]: pgmap v44: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 6.7 MiB/s wr, 673 op/s 2024-09-20T23:50:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:11.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:12.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 167089 10.86 MB/sec execute 236 sec latency 61.543 ms 2024-09-20T23:50:13.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 167310 10.84 MB/sec execute 237 sec latency 52.780 ms 2024-09-20T23:50:13.406 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:13 smithi137 ceph-mon[65321]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-09-20T23:50:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:13 smithi144 ceph-mon[51829]: pgmap v45: 97 pgs: 97 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 454 op/s 2024-09-20T23:50:14.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 167618 10.82 MB/sec execute 238 sec latency 65.341 ms 2024-09-20T23:50:15.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 167976 10.82 MB/sec execute 239 sec latency 64.807 ms 2024-09-20T23:50:15.364 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:15 smithi137 ceph-mon[65321]: pgmap v46: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-09-20T23:50:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:15 smithi144 ceph-mon[51829]: pgmap v46: 97 pgs: 97 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 5.9 MiB/s wr, 617 op/s 2024-09-20T23:50:16.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 168233 10.80 MB/sec execute 240 sec latency 83.704 ms 2024-09-20T23:50:17.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 168573 10.78 MB/sec execute 241 sec latency 38.542 ms 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 479 op/s 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]: dispatch 2024-09-20T23:50:17.940 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]': finished 2024-09-20T23:50:17.941 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi144"}]: dispatch 2024-09-20T23:50:17.941 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi144"}]': finished 2024-09-20T23:50:17.941 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T23:50:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: pgmap v47: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.1 MiB/s wr, 479 op/s 2024-09-20T23:50:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:18.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi137"}]': finished 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi144"}]: dispatch 2024-09-20T23:50:18.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash.smithi144"}]': finished 2024-09-20T23:50:18.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T23:50:18.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 168857 10.76 MB/sec execute 242 sec latency 59.485 ms 2024-09-20T23:50:18.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:18 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all crash 2024-09-20T23:50:18.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:18 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T23:50:18.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:18 smithi137 ceph-mon[65321]: Upgrade: osd.0 is safe to restart 2024-09-20T23:50:18.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:18 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all crash 2024-09-20T23:50:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:18 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["0"], "max": 16}]: dispatch 2024-09-20T23:50:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:18 smithi144 ceph-mon[51829]: Upgrade: osd.0 is safe to restart 2024-09-20T23:50:19.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:19.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 169199 10.77 MB/sec execute 243 sec latency 42.492 ms 2024-09-20T23:50:20.018 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:19 smithi137 ceph-mon[65321]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 MiB/s wr, 450 op/s 2024-09-20T23:50:20.018 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:20.018 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:50:20.018 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:19 smithi144 ceph-mon[51829]: pgmap v48: 97 pgs: 97 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.5 MiB/s wr, 450 op/s 2024-09-20T23:50:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-20T23:50:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:20.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 169494 10.76 MB/sec execute 244 sec latency 54.702 ms 2024-09-20T23:50:21.039 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:20 smithi137 ceph-mon[65321]: Upgrade: Updating osd.0 2024-09-20T23:50:21.039 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:20 smithi137 ceph-mon[65321]: Deploying daemon osd.0 on smithi137 2024-09-20T23:50:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:20 smithi144 ceph-mon[51829]: Upgrade: Updating osd.0 2024-09-20T23:50:21.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:20 smithi144 ceph-mon[51829]: Deploying daemon osd.0 on smithi137 2024-09-20T23:50:21.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 169855 10.79 MB/sec execute 245 sec latency 53.231 ms 2024-09-20T23:50:22.058 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:21 smithi137 ceph-mon[65321]: pgmap v49: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-09-20T23:50:22.058 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:21 smithi137 ceph-mon[65321]: mgrmap e42: smithi137.eurecx(active, since 92s), standbys: smithi144.ygsngz 2024-09-20T23:50:22.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:21 smithi144 ceph-mon[51829]: pgmap v49: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.6 KiB/s rd, 6.2 MiB/s wr, 633 op/s 2024-09-20T23:50:22.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:21 smithi144 ceph-mon[51829]: mgrmap e42: smithi137.eurecx(active, since 92s), standbys: smithi144.ygsngz 2024-09-20T23:50:22.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 170193 10.77 MB/sec execute 246 sec latency 47.675 ms 2024-09-20T23:50:23.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 170412 10.75 MB/sec execute 247 sec latency 60.362 ms 2024-09-20T23:50:24.086 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:23 smithi137 ceph-mon[65321]: pgmap v50: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 387 op/s 2024-09-20T23:50:24.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 170655 10.72 MB/sec execute 248 sec latency 65.830 ms 2024-09-20T23:50:24.340 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:24 smithi137 systemd[1]: Stopping Ceph osd.0 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:50:24.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:23 smithi144 ceph-mon[51829]: pgmap v50: 97 pgs: 97 active+clean; 342 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 3.8 MiB/s wr, 387 op/s 2024-09-20T23:50:24.596 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:24 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[34146]: 2024-09-20T23:50:24.484+0000 7f6492162700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:50:24.596 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:24 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[34146]: 2024-09-20T23:50:24.484+0000 7f6492162700 -1 osd.0 73 *** Got signal Terminated *** 2024-09-20T23:50:24.597 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:24 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[34146]: 2024-09-20T23:50:24.484+0000 7f6492162700 -1 osd.0 73 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:50:25.109 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:24 smithi137 ceph-mon[65321]: osd.0 marked itself down and dead 2024-09-20T23:50:25.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 170947 10.71 MB/sec execute 249 sec latency 266.334 ms 2024-09-20T23:50:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:24 smithi144 ceph-mon[51829]: osd.0 marked itself down and dead 2024-09-20T23:50:25.928 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:25 smithi137 podman[70682]: 2024-09-20 23:50:25.656981843 +0000 UTC m=+1.304501238 container died 7f9d63ee718456f84937fcec6bc77566ec8955b93f6219e2baaeb828df3dbc2c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, io.buildah.version=1.19.8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, release=754, GIT_BRANCH=HEAD, GIT_CLEAN=True, architecture=x86_64, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, ceph=True, version=8) 2024-09-20T23:50:26.202 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:25 smithi137 ceph-mon[65321]: pgmap v51: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 549 op/s 2024-09-20T23:50:26.202 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:25 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:50:26.202 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:25 smithi137 ceph-mon[65321]: osdmap e74: 8 total, 7 up, 8 in 2024-09-20T23:50:26.202 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:26 smithi137 podman[70682]: 2024-09-20 23:50:26.116356725 +0000 UTC m=+1.763876125 container cleanup 7f9d63ee718456f84937fcec6bc77566ec8955b93f6219e2baaeb828df3dbc2c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, version=8, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, distribution-scope=public, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_CLEAN=True) 2024-09-20T23:50:26.202 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:26 smithi137 bash[70682]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0 2024-09-20T23:50:26.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 171147 10.68 MB/sec execute 250 sec latency 1002.362 ms 2024-09-20T23:50:26.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:25 smithi144 ceph-mon[51829]: pgmap v51: 97 pgs: 97 active+clean; 351 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 549 op/s 2024-09-20T23:50:26.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:25 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:50:26.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:25 smithi144 ceph-mon[51829]: osdmap e74: 8 total, 7 up, 8 in 2024-09-20T23:50:26.452 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:26 smithi137 podman[70698]: 2024-09-20 23:50:26.418999962 +0000 UTC m=+0.755851566 container remove 7f9d63ee718456f84937fcec6bc77566ec8955b93f6219e2baaeb828df3dbc2c (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.buildah.version=1.19.8, release=754, ceph=True, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc.) 2024-09-20T23:50:26.973 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:50:27.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:26 smithi137 ceph-mon[65321]: osdmap e75: 8 total, 7 up, 8 in 2024-09-20T23:50:27.004 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:26 smithi137 podman[70801]: 2024-09-20 23:50:26.883990007 +0000 UTC m=+0.024170538 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:50:27.004 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:26 smithi137 podman[70801]: 2024-09-20 23:50:26.995132714 +0000 UTC m=+0.135313238 container create 41ce4bb26746af7e14de3bac070d32d0f014a49c9c4cdff36bcb030ddd0213d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T23:50:27.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 171442 10.68 MB/sec execute 251 sec latency 64.678 ms 2024-09-20T23:50:27.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:26 smithi144 ceph-mon[51829]: osdmap e75: 8 total, 7 up, 8 in 2024-09-20T23:50:27.523 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:27 smithi137 podman[70801]: 2024-09-20 23:50:27.409835747 +0000 UTC m=+0.550016289 container init 41ce4bb26746af7e14de3bac070d32d0f014a49c9c4cdff36bcb030ddd0213d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T23:50:27.523 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:27 smithi137 podman[70801]: 2024-09-20 23:50:27.424292418 +0000 UTC m=+0.564472948 container start 41ce4bb26746af7e14de3bac070d32d0f014a49c9c4cdff36bcb030ddd0213d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:50:27.776 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:27 smithi137 podman[70801]: 2024-09-20 23:50:27.583354935 +0000 UTC m=+0.723535461 container attach 41ce4bb26746af7e14de3bac070d32d0f014a49c9c4cdff36bcb030ddd0213d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T23:50:27.777 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:27 smithi137 podman[70801]: 2024-09-20 23:50:27.752004517 +0000 UTC m=+0.892185040 container died 41ce4bb26746af7e14de3bac070d32d0f014a49c9c4cdff36bcb030ddd0213d3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (3m) 17s ago 12m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (33s) 17s ago 12m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (27s) 23s ago 12m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (3m) 17s ago 12m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:50:27.891 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (6m) 17s ago 6m 40.7M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (6m) 23s ago 6m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (2m) 17s ago 13m 493M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (2m) 23s ago 12m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (107s) 17s ago 14m 42.2M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (84s) 23s ago 11m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (6m) 17s ago 6m 264M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (6m) 23s ago 6m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (3m) 17s ago 12m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (3m) 23s ago 11m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (11m) 17s ago 11m 319M 1936M 17.2.0 e1d6a67b021e 7f9d63ee7184 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (10m) 17s ago 10m 486M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (10m) 17s ago 10m 348M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (9m) 17s ago 9m 515M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:50:27.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (9m) 23s ago 9m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:50:27.893 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (9m) 23s ago 9m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:50:27.893 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (8m) 23s ago 8m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:50:27.893 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (8m) 23s ago 8m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:50:27.893 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (2m) 17s ago 11m 60.1M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:50:28.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:27 smithi137 ceph-mon[65321]: pgmap v54: 97 pgs: 13 stale+active+clean, 84 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-09-20T23:50:28.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:27 smithi137 ceph-mon[65321]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:27 smithi137 ceph-mon[65321]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:27 smithi137 ceph-mon[65321]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.243 INFO:teuthology.orchestra.run.smithi137.stdout: 5 171768 10.66 MB/sec execute 252 sec latency 69.572 ms 2024-09-20T23:50:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:27 smithi144 ceph-mon[51829]: pgmap v54: 97 pgs: 13 stale+active+clean, 84 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.5 MiB/s wr, 544 op/s 2024-09-20T23:50:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:27 smithi144 ceph-mon[51829]: from='client.44137 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:27 smithi144 ceph-mon[51829]: from='client.34184 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:27 smithi144 ceph-mon[51829]: from='client.34188 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:50:28.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:50:28.597 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:50:28.597 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-09-20T23:50:28.597 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T23:50:28.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:50:28.597 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:50:28.787 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:28 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.0.service: Deactivated successfully. 2024-09-20T23:50:28.787 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:28 smithi137 systemd[1]: Stopped Ceph osd.0 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:50:28.787 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:28 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.0.service: Consumed 2min 47.232s CPU time. 2024-09-20T23:50:29.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:28 smithi137 ceph-mon[65321]: pgmap v55: 97 pgs: 13 stale+active+clean, 84 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-09-20T23:50:29.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:28 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3528575861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:50:29.164 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "6/23 daemons upgraded", 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:50:29.165 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:50:29.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 172073 10.64 MB/sec execute 253 sec latency 60.729 ms 2024-09-20T23:50:29.291 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:29 smithi137 systemd[1]: Starting Ceph osd.0 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:50:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:28 smithi144 ceph-mon[51829]: pgmap v55: 97 pgs: 13 stale+active+clean, 84 active+clean; 348 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.9 KiB/s rd, 3.0 MiB/s wr, 269 op/s 2024-09-20T23:50:29.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:28 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3528575861' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:29.543 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:29 smithi137 podman[71018]: 2024-09-20 23:50:29.322688773 +0000 UTC m=+0.022937423 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:50:29.543 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:29 smithi137 podman[71018]: 2024-09-20 23:50:29.45119046 +0000 UTC m=+0.151439088 container create 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.37.2) 2024-09-20T23:50:29.836 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down 2024-09-20T23:50:29.836 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:50:29.836 INFO:teuthology.orchestra.run.smithi137.stdout: osd.0 (root=default,host=smithi137) is down 2024-09-20T23:50:30.046 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:29 smithi137 ceph-mon[65321]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:30.047 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:29 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2874635247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:50:30.047 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:29 smithi137 podman[71018]: 2024-09-20 23:50:29.929829202 +0000 UTC m=+0.630077848 container init 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T23:50:30.047 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:29 smithi137 podman[71018]: 2024-09-20 23:50:29.943496385 +0000 UTC m=+0.643745014 container start 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-09-20T23:50:30.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 172472 10.64 MB/sec execute 254 sec latency 28.589 ms 2024-09-20T23:50:30.298 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:30 smithi137 podman[71018]: 2024-09-20 23:50:30.115045719 +0000 UTC m=+0.815294358 container attach 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-20T23:50:30.298 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:30 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:50:30.298 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:30 smithi137 bash[71018]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:50:30.298 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:30 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:50:30.298 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:30 smithi137 bash[71018]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:50:30.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:29 smithi144 ceph-mon[51829]: from='client.34196 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:50:30.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:29 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2874635247' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:50:31.054 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:30 smithi137 ceph-mon[65321]: pgmap v56: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.2 MiB/s wr, 533 op/s; 208/1584 objects degraded (13.131%) 2024-09-20T23:50:31.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 173056 10.69 MB/sec execute 255 sec latency 52.632 ms 2024-09-20T23:50:31.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:30 smithi144 ceph-mon[51829]: pgmap v56: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 5.2 MiB/s wr, 533 op/s; 208/1584 objects degraded (13.131%) 2024-09-20T23:50:31.813 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:31 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T23:50:31.813 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:31 smithi137 bash[71018]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T23:50:31.813 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:31 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T23:50:31.813 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:31 smithi137 bash[71018]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T23:50:32.063 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:31 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 208/1584 objects degraded (13.131%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:32.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 173468 10.71 MB/sec execute 256 sec latency 59.061 ms 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 bash[71018]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 bash[71018]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 bash[71018]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 bash[71018]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate[71050]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 bash[71018]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-20T23:50:32.315 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:32 smithi137 podman[71018]: 2024-09-20 23:50:32.110960539 +0000 UTC m=+2.811209169 container died 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, ceph=True) 2024-09-20T23:50:32.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:31 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 208/1584 objects degraded (13.131%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:33.093 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:33 smithi137 ceph-mon[65321]: pgmap v57: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 MiB/s wr, 289 op/s; 208/1584 objects degraded (13.131%) 2024-09-20T23:50:33.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 173719 10.69 MB/sec execute 257 sec latency 50.289 ms 2024-09-20T23:50:33.343 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:33 smithi144 ceph-mon[51829]: pgmap v57: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 3.7 KiB/s rd, 2.7 MiB/s wr, 289 op/s; 208/1584 objects degraded (13.131%) 2024-09-20T23:50:33.344 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 podman[71203]: 2024-09-20 23:50:33.203146781 +0000 UTC m=+1.085015947 container remove 12cee6ca761f464d11dd9cf2ba2dc5bdc4d77a60812fe1b43213429dcaa6208f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-activate, GIT_CLEAN=True, ceph=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux ) 2024-09-20T23:50:33.619 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 podman[71262]: 2024-09-20 23:50:33.362357875 +0000 UTC m=+0.027746191 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:50:33.619 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 podman[71262]: 2024-09-20 23:50:33.487684141 +0000 UTC m=+0.153072442 container create a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T23:50:33.873 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 podman[71262]: 2024-09-20 23:50:33.756943262 +0000 UTC m=+0.422331573 container init a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-20T23:50:33.873 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 podman[71262]: 2024-09-20 23:50:33.765106559 +0000 UTC m=+0.430494862 container start a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T23:50:33.873 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 ceph-osd[71281]: -- 172.21.15.137:0/2491604892 <== mon.0 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5631b8eaa1e0 con 0x5631b826e000 2024-09-20T23:50:33.873 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 bash[71262]: a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa 2024-09-20T23:50:33.873 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:33 smithi137 systemd[1]: Started Ceph osd.0 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:50:34.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 173930 10.66 MB/sec execute 258 sec latency 63.165 ms 2024-09-20T23:50:34.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:34.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:34.590 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.590 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:34.590 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.591 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:34.591 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:35.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 174157 10.63 MB/sec execute 259 sec latency 47.861 ms 2024-09-20T23:50:35.928 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:35 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:50:35.856+0000 7f13ad9b8640 -1 Falling back to public interface 2024-09-20T23:50:36.194 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:35 smithi137 ceph-mon[65321]: pgmap v58: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 422 op/s; 202/1632 objects degraded (12.377%) 2024-09-20T23:50:36.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 174469 10.61 MB/sec execute 260 sec latency 62.665 ms 2024-09-20T23:50:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:35 smithi144 ceph-mon[51829]: pgmap v58: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 340 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.4 KiB/s rd, 4.1 MiB/s wr, 422 op/s; 202/1632 objects degraded (12.377%) 2024-09-20T23:50:37.234 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:36 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 202/1632 objects degraded (12.377%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:37.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 174797 10.61 MB/sec execute 261 sec latency 37.553 ms 2024-09-20T23:50:37.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:36 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 202/1632 objects degraded (12.377%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:37.991 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:37 smithi137 ceph-mon[65321]: pgmap v59: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 467 op/s; 213/1674 objects degraded (12.724%) 2024-09-20T23:50:38.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 175239 10.61 MB/sec execute 262 sec latency 57.930 ms 2024-09-20T23:50:38.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:37 smithi144 ceph-mon[51829]: pgmap v59: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 467 op/s; 213/1674 objects degraded (12.724%) 2024-09-20T23:50:39.009 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:38 smithi137 ceph-mon[65321]: pgmap v60: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 397 op/s; 213/1674 objects degraded (12.724%) 2024-09-20T23:50:39.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 175675 10.60 MB/sec execute 263 sec latency 77.680 ms 2024-09-20T23:50:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:38 smithi144 ceph-mon[51829]: pgmap v60: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 346 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.0 MiB/s wr, 397 op/s; 213/1674 objects degraded (12.724%) 2024-09-20T23:50:40.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 176014 10.59 MB/sec execute 264 sec latency 79.106 ms 2024-09-20T23:50:41.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 176415 10.61 MB/sec execute 265 sec latency 119.134 ms 2024-09-20T23:50:41.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:41 smithi137 ceph-mon[65321]: pgmap v61: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 612 op/s; 209/1629 objects degraded (12.830%) 2024-09-20T23:50:41.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:41 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 209/1629 objects degraded (12.830%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:41 smithi144 ceph-mon[51829]: pgmap v61: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.8 MiB/s wr, 612 op/s; 209/1629 objects degraded (12.830%) 2024-09-20T23:50:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:41 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 209/1629 objects degraded (12.830%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:42.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 176790 10.63 MB/sec execute 266 sec latency 106.038 ms 2024-09-20T23:50:43.116 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.117 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.117 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-mon[65321]: pgmap v62: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 436 op/s; 209/1629 objects degraded (12.830%) 2024-09-20T23:50:43.117 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.117 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 177107 10.63 MB/sec execute 267 sec latency 87.210 ms 2024-09-20T23:50:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:43 smithi144 ceph-mon[51829]: pgmap v62: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 336 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.4 MiB/s wr, 436 op/s; 209/1629 objects degraded (12.830%) 2024-09-20T23:50:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:43 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:43.895 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:43 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:50:43.859+0000 7f13ad9b8640 -1 osd.0 0 read_superblock omap replica is missing. 2024-09-20T23:50:44.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 177398 10.62 MB/sec execute 268 sec latency 42.573 ms 2024-09-20T23:50:44.923 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:50:44.820+0000 7f13ad9b8640 -1 osd.0 73 log_to_monitors true 2024-09-20T23:50:45.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 177639 10.59 MB/sec execute 269 sec latency 55.255 ms 2024-09-20T23:50:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:45 smithi144 ceph-mon[51829]: pgmap v63: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 542 op/s; 207/1641 objects degraded (12.614%) 2024-09-20T23:50:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:45 smithi144 ceph-mon[51829]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T23:50:45.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:45 smithi137 ceph-mon[65321]: pgmap v63: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 339 MiB data, 3.7 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 5.9 MiB/s wr, 542 op/s; 207/1641 objects degraded (12.614%) 2024-09-20T23:50:45.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:45 smithi137 ceph-mon[65321]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-20T23:50:46.007 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:50:45 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:50:45.797+0000 7f13a175b640 -1 osd.0 73 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:50:46.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 177894 10.57 MB/sec execute 270 sec latency 204.332 ms 2024-09-20T23:50:46.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:46 smithi137 ceph-mon[65321]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T23:50:46.509 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:46 smithi137 ceph-mon[65321]: osdmap e76: 8 total, 7 up, 8 in 2024-09-20T23:50:46.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:46 smithi137 ceph-mon[65321]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:50:46.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:46 smithi137 ceph-mon[65321]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' 2024-09-20T23:50:46.510 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:46 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 207/1641 objects degraded (12.614%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:46 smithi144 ceph-mon[51829]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-20T23:50:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:46 smithi144 ceph-mon[51829]: osdmap e76: 8 total, 7 up, 8 in 2024-09-20T23:50:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:46 smithi144 ceph-mon[51829]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:50:46.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:46 smithi144 ceph-mon[51829]: from='osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846]' entity='osd.0' 2024-09-20T23:50:46.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:46 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 207/1641 objects degraded (12.614%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:47.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178068 10.54 MB/sec execute 271 sec latency 478.785 ms 2024-09-20T23:50:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:47 smithi137 ceph-mon[65321]: pgmap v65: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 510 op/s; 213/1686 objects degraded (12.633%) 2024-09-20T23:50:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:47 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:50:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:47 smithi137 ceph-mon[65321]: osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846] boot 2024-09-20T23:50:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:47 smithi137 ceph-mon[65321]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T23:50:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:50:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:47 smithi144 ceph-mon[51829]: pgmap v65: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 5.2 MiB/s wr, 510 op/s; 213/1686 objects degraded (12.633%) 2024-09-20T23:50:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:47 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:50:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:47 smithi144 ceph-mon[51829]: osd.0 [v2:172.21.15.137:6802/1912811846,v1:172.21.15.137:6803/1912811846] boot 2024-09-20T23:50:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:47 smithi144 ceph-mon[51829]: osdmap e77: 8 total, 8 up, 8 in 2024-09-20T23:50:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-20T23:50:48.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178068 10.50 MB/sec execute 272 sec latency 1478.811 ms 2024-09-20T23:50:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T23:50:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:48.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: osdmap e78: 8 total, 8 up, 8 in 2024-09-20T23:50:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:50:49.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:50:49.244 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178131 10.47 MB/sec execute 273 sec latency 2220.874 ms 2024-09-20T23:50:49.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:49 smithi137 ceph-mon[65321]: pgmap v68: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 421 op/s; 213/1686 objects degraded (12.633%) 2024-09-20T23:50:49.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:49 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:50:49.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:49 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-20T23:50:49.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:49 smithi137 ceph-mon[65321]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T23:50:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:49 smithi144 ceph-mon[51829]: pgmap v68: 97 pgs: 7 active+undersized, 29 active+undersized+degraded, 61 active+clean; 345 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.1 MiB/s wr, 421 op/s; 213/1686 objects degraded (12.633%) 2024-09-20T23:50:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:49 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:50:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:49 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (9 PGs are or would become offline) 2024-09-20T23:50:50.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:49 smithi144 ceph-mon[51829]: osdmap e79: 8 total, 8 up, 8 in 2024-09-20T23:50:50.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178448 10.48 MB/sec execute 274 sec latency 172.264 ms 2024-09-20T23:50:51.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178488 10.45 MB/sec execute 275 sec latency 1062.986 ms 2024-09-20T23:50:51.349 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:51 smithi137 ceph-mon[65321]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T23:50:51.350 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:51 smithi137 ceph-mon[65321]: pgmap v71: 97 pgs: 1 active+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized, 4 active+undersized+degraded, 76 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.0 MiB/s wr, 325 op/s; 243/1392 objects degraded (17.457%); 1024 KiB/s, 9 objects/s recovering 2024-09-20T23:50:51.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:51 smithi144 ceph-mon[51829]: osdmap e80: 8 total, 8 up, 8 in 2024-09-20T23:50:51.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:51 smithi144 ceph-mon[51829]: pgmap v71: 97 pgs: 1 active+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 5 active+undersized, 4 active+undersized+degraded, 76 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.0 MiB/s wr, 325 op/s; 243/1392 objects degraded (17.457%); 1024 KiB/s, 9 objects/s recovering 2024-09-20T23:50:52.131 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:52 smithi137 ceph-mon[65321]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T23:50:52.131 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:52 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 243/1392 objects degraded (17.457%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:52.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 178779 10.43 MB/sec execute 276 sec latency 997.843 ms 2024-09-20T23:50:52.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:52 smithi144 ceph-mon[51829]: osdmap e81: 8 total, 8 up, 8 in 2024-09-20T23:50:52.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:52 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 243/1392 objects degraded (17.457%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:53.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 179048 10.40 MB/sec execute 277 sec latency 952.978 ms 2024-09-20T23:50:53.254 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:53 smithi137 ceph-mon[65321]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T23:50:53.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:53 smithi137 ceph-mon[65321]: pgmap v74: 97 pgs: 1 active+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.0 MiB/s wr, 329 op/s; 209/1380 objects degraded (15.145%); 1024 KiB/s, 17 objects/s recovering 2024-09-20T23:50:53.255 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:53 smithi137 ceph-mon[65321]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T23:50:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:53 smithi144 ceph-mon[51829]: osdmap e82: 8 total, 8 up, 8 in 2024-09-20T23:50:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:53 smithi144 ceph-mon[51829]: pgmap v74: 97 pgs: 1 active+undersized+remapped, 8 active+recovery_wait+undersized+degraded+remapped, 2 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 85 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 3.0 MiB/s wr, 329 op/s; 209/1380 objects degraded (15.145%); 1024 KiB/s, 17 objects/s recovering 2024-09-20T23:50:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:53 smithi144 ceph-mon[51829]: osdmap e83: 8 total, 8 up, 8 in 2024-09-20T23:50:54.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 179467 10.40 MB/sec execute 278 sec latency 933.342 ms 2024-09-20T23:50:55.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 179900 10.41 MB/sec execute 279 sec latency 28.793 ms 2024-09-20T23:50:55.294 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:55 smithi137 ceph-mon[65321]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T23:50:55.295 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:55 smithi137 ceph-mon[65321]: pgmap v77: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.2 MiB/s wr, 255 op/s; 162/1491 objects degraded (10.865%); 1.0 MiB/s, 19 objects/s recovering 2024-09-20T23:50:55.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:55 smithi144 ceph-mon[51829]: osdmap e84: 8 total, 8 up, 8 in 2024-09-20T23:50:55.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:55 smithi144 ceph-mon[51829]: pgmap v77: 97 pgs: 6 active+recovery_wait+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 1 active+recovering+undersized+degraded+remapped, 89 active+clean; 333 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.2 MiB/s wr, 255 op/s; 162/1491 objects degraded (10.865%); 1.0 MiB/s, 19 objects/s recovering 2024-09-20T23:50:56.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 180409 10.44 MB/sec execute 280 sec latency 921.281 ms 2024-09-20T23:50:56.311 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:56 smithi137 ceph-mon[65321]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T23:50:56.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:56 smithi144 ceph-mon[51829]: osdmap e85: 8 total, 8 up, 8 in 2024-09-20T23:50:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 180862 10.46 MB/sec execute 281 sec latency 60.999 ms 2024-09-20T23:50:57.321 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:57 smithi137 ceph-mon[65321]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T23:50:57.322 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:57 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 162/1491 objects degraded (10.865%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:57.322 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:57 smithi137 ceph-mon[65321]: pgmap v80: 97 pgs: 1 active+recovering+degraded, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 MiB/s wr, 452 op/s; 1/1620 objects degraded (0.062%); 2.7 MiB/s, 50 objects/s recovering 2024-09-20T23:50:57.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:57 smithi144 ceph-mon[51829]: osdmap e86: 8 total, 8 up, 8 in 2024-09-20T23:50:57.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:57 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 162/1491 objects degraded (10.865%), 8 pgs degraded (PG_DEGRADED) 2024-09-20T23:50:57.446 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:57 smithi144 ceph-mon[51829]: pgmap v80: 97 pgs: 1 active+recovering+degraded, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.2 KiB/s rd, 4.2 MiB/s wr, 452 op/s; 1/1620 objects degraded (0.062%); 2.7 MiB/s, 50 objects/s recovering 2024-09-20T23:50:58.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 181282 10.45 MB/sec execute 282 sec latency 36.836 ms 2024-09-20T23:50:59.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 181902 10.47 MB/sec execute 283 sec latency 28.871 ms 2024-09-20T23:50:59.336 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:50:59 smithi137 ceph-mon[65321]: pgmap v81: 97 pgs: 1 active+recovering+degraded, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.3 MiB/s wr, 356 op/s; 1/1620 objects degraded (0.062%); 2.2 MiB/s, 39 objects/s recovering 2024-09-20T23:50:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:50:59 smithi144 ceph-mon[51829]: pgmap v81: 97 pgs: 1 active+recovering+degraded, 1 active+undersized+remapped, 95 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.3 MiB/s wr, 356 op/s; 1/1620 objects degraded (0.062%); 2.2 MiB/s, 39 objects/s recovering 2024-09-20T23:51:00.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 182628 10.49 MB/sec execute 284 sec latency 43.356 ms 2024-09-20T23:51:00.345 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:00 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1620 objects degraded (0.062%), 1 pg degraded) 2024-09-20T23:51:00.346 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:00 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:51:00.447 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:51:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:00 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1620 objects degraded (0.062%), 1 pg degraded) 2024-09-20T23:51:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:00 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:51:01.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 183279 10.50 MB/sec execute 285 sec latency 33.781 ms 2024-09-20T23:51:01.370 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:01 smithi137 ceph-mon[65321]: pgmap v82: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 8.4 MiB/s wr, 850 op/s; 1.8 MiB/s, 34 objects/s recovering 2024-09-20T23:51:01.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:01 smithi137 ceph-mon[65321]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:01.499 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:01.499 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (4m) 19s ago 13m 27.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:51:01.499 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (67s) 19s ago 13m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (61s) 56s ago 12m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (4m) 19s ago 13m 55.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (7m) 19s ago 7m 40.7M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (7m) 56s ago 7m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (3m) 19s ago 14m 496M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (2m) 56s ago 12m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (2m) 19s ago 14m 45.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (117s) 56s ago 12m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (6m) 19s ago 6m 270M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (6m) 56s ago 6m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:51:01.500 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 19s ago 12m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (3m) 56s ago 12m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (28s) 19s ago 11m 11.4M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (11m) 19s ago 11m 531M 1936M 17.2.0 e1d6a67b021e 9c8b7d1321ac 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (10m) 19s ago 10m 380M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (10m) 19s ago 10m 538M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (10m) 56s ago 9m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (9m) 56s ago 9m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (9m) 56s ago 9m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (8m) 56s ago 8m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:51:01.501 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (2m) 19s ago 12m 64.4M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:51:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:01 smithi144 ceph-mon[51829]: pgmap v82: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.8 KiB/s rd, 8.4 MiB/s wr, 850 op/s; 1.8 MiB/s, 34 objects/s recovering 2024-09-20T23:51:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:01 smithi144 ceph-mon[51829]: from='client.34208 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 9, 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 5 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:51:02.012 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:51:02.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 183515 10.50 MB/sec execute 286 sec latency 68.808 ms 2024-09-20T23:51:02.403 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:02 smithi137 ceph-mon[65321]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:02.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:02 smithi137 ceph-mon[65321]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:02.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:02 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3796624881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:02.497 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:51:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:51:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:51:02.497 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "7/23 daemons upgraded", 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:51:02.498 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:51:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:02 smithi144 ceph-mon[51829]: from='client.34212 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:02 smithi144 ceph-mon[51829]: from='client.44169 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:02 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3796624881' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:03.201 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:51:03.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 183826 10.51 MB/sec execute 287 sec latency 64.391 ms 2024-09-20T23:51:03.420 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: pgmap v83: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.7 MiB/s wr, 543 op/s; 873 KiB/s, 20 objects/s recovering 2024-09-20T23:51:03.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:03.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1292523801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:51:03.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:51:03.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:03.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:03.593 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: pgmap v83: 97 pgs: 97 active+clean; 347 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 5.7 MiB/s wr, 543 op/s; 873 KiB/s, 20 objects/s recovering 2024-09-20T23:51:03.594 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: from='client.44175 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:03.594 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/1292523801' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:51:03.594 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:51:03.594 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:03.594 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:04.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 184164 10.50 MB/sec execute 288 sec latency 62.530 ms 2024-09-20T23:51:04.454 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:04 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:51:04.454 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:04 smithi137 ceph-mon[65321]: Upgrade: osd.1 is safe to restart 2024-09-20T23:51:04.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:04 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["1"], "max": 16}]: dispatch 2024-09-20T23:51:04.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:04 smithi144 ceph-mon[51829]: Upgrade: osd.1 is safe to restart 2024-09-20T23:51:05.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 184442 10.50 MB/sec execute 289 sec latency 54.825 ms 2024-09-20T23:51:05.442 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:05 smithi137 ceph-mon[65321]: pgmap v84: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 634 op/s; 0 B/s, 12 objects/s recovering 2024-09-20T23:51:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:05 smithi144 ceph-mon[51829]: pgmap v84: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 6.6 MiB/s wr, 634 op/s; 0 B/s, 12 objects/s recovering 2024-09-20T23:51:06.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 184621 10.48 MB/sec execute 290 sec latency 60.685 ms 2024-09-20T23:51:06.788 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:06 smithi137 ceph-mon[65321]: Upgrade: Updating osd.1 2024-09-20T23:51:06.789 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:06 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:06.789 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:06 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:51:06.789 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:06 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:06.789 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:06 smithi137 ceph-mon[65321]: Deploying daemon osd.1 on smithi137 2024-09-20T23:51:06.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:06 smithi144 ceph-mon[51829]: Upgrade: Updating osd.1 2024-09-20T23:51:06.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:06 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:06.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:06 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-20T23:51:06.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:06 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:06.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:06 smithi144 ceph-mon[51829]: Deploying daemon osd.1 on smithi137 2024-09-20T23:51:07.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 184900 10.46 MB/sec execute 291 sec latency 88.753 ms 2024-09-20T23:51:07.798 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:07 smithi137 ceph-mon[65321]: pgmap v85: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 663 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:07 smithi144 ceph-mon[51829]: pgmap v85: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 6.9 MiB/s wr, 663 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:08.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 185312 10.46 MB/sec execute 292 sec latency 66.603 ms 2024-09-20T23:51:09.245 INFO:teuthology.orchestra.run.smithi137.stdout: 5 185652 10.45 MB/sec execute 293 sec latency 40.681 ms 2024-09-20T23:51:09.334 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:09 smithi137 ceph-mon[65321]: pgmap v86: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 555 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:09.335 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:09 smithi144 ceph-mon[51829]: pgmap v86: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 5.8 MiB/s wr, 555 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:09.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:10.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 185950 10.45 MB/sec execute 294 sec latency 83.733 ms 2024-09-20T23:51:10.613 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:10 smithi137 systemd[1]: Stopping Ceph osd.1 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:51:11.121 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:10 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[38146]: 2024-09-20T23:51:10.912+0000 7f7b6d956700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:51:11.121 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:10 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[38146]: 2024-09-20T23:51:10.912+0000 7f7b6d956700 -1 osd.1 86 *** Got signal Terminated *** 2024-09-20T23:51:11.121 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:10 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[38146]: 2024-09-20T23:51:10.912+0000 7f7b6d956700 -1 osd.1 86 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:51:11.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 186340 10.43 MB/sec execute 295 sec latency 84.343 ms 2024-09-20T23:51:11.373 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:11 smithi137 ceph-mon[65321]: pgmap v87: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 732 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:11.374 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:11 smithi137 ceph-mon[65321]: osd.1 marked itself down and dead 2024-09-20T23:51:11.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:11 smithi144 ceph-mon[51829]: pgmap v87: 97 pgs: 97 active+clean; 338 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 7.5 MiB/s wr, 732 op/s; 0 B/s, 0 objects/s recovering 2024-09-20T23:51:11.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:11 smithi144 ceph-mon[51829]: osd.1 marked itself down and dead 2024-09-20T23:51:12.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 186485 10.41 MB/sec execute 296 sec latency 958.775 ms 2024-09-20T23:51:12.255 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:11 smithi137 podman[73413]: 2024-09-20 23:51:11.938027039 +0000 UTC m=+1.176167496 container died 9c8b7d1321ac6abc4f9639d3a93a873bc2e2c7f25faa198cbd5dbc0dce852672 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, io.openshift.tags=base centos centos-stream, vcs-type=git, version=8, distribution-scope=public, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=) 2024-09-20T23:51:12.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:12 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:51:12.507 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:12 smithi137 ceph-mon[65321]: osdmap e87: 8 total, 7 up, 8 in 2024-09-20T23:51:12.507 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:12 smithi137 podman[73413]: 2024-09-20 23:51:12.361736738 +0000 UTC m=+1.599877186 container cleanup 9c8b7d1321ac6abc4f9639d3a93a873bc2e2c7f25faa198cbd5dbc0dce852672 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, ceph=True, RELEASE=HEAD, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, vendor=Red Hat, Inc., name=centos-stream, release=754, GIT_BRANCH=HEAD, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, version=8, GIT_CLEAN=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T23:51:12.507 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:12 smithi137 bash[73413]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1 2024-09-20T23:51:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:12 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:51:12.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:12 smithi144 ceph-mon[51829]: osdmap e87: 8 total, 7 up, 8 in 2024-09-20T23:51:12.759 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:12 smithi137 podman[73427]: 2024-09-20 23:51:12.672118524 +0000 UTC m=+0.729020673 container remove 9c8b7d1321ac6abc4f9639d3a93a873bc2e2c7f25faa198cbd5dbc0dce852672 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., architecture=x86_64, name=centos-stream, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git) 2024-09-20T23:51:13.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 186789 10.40 MB/sec execute 297 sec latency 1036.725 ms 2024-09-20T23:51:13.276 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.16184169 +0000 UTC m=+0.018856286 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:51:13.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:13 smithi137 ceph-mon[65321]: pgmap v89: 97 pgs: 14 stale+active+clean, 83 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.5 MiB/s wr, 542 op/s 2024-09-20T23:51:13.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:13 smithi137 ceph-mon[65321]: osdmap e88: 8 total, 7 up, 8 in 2024-09-20T23:51:13.538 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.307203556 +0000 UTC m=+0.164218149 container create 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T23:51:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:13 smithi144 ceph-mon[51829]: pgmap v89: 97 pgs: 14 stale+active+clean, 83 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.4 KiB/s rd, 5.5 MiB/s wr, 542 op/s 2024-09-20T23:51:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:13 smithi144 ceph-mon[51829]: osdmap e88: 8 total, 7 up, 8 in 2024-09-20T23:51:13.788 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.677149298 +0000 UTC m=+0.534163888 container init 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-20T23:51:13.789 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.685067902 +0000 UTC m=+0.542082499 container start 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-20T23:51:14.039 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.847775113 +0000 UTC m=+0.704789710 container attach 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T23:51:14.040 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:13 smithi137 podman[73509]: 2024-09-20 23:51:13.994615578 +0000 UTC m=+0.851630173 container died 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_CLEAN=True) 2024-09-20T23:51:14.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 187028 10.39 MB/sec execute 298 sec latency 48.056 ms 2024-09-20T23:51:14.543 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:14 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 122/1617 objects degraded (7.545%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:14 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 122/1617 objects degraded (7.545%), 17 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:15.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 187402 10.40 MB/sec execute 299 sec latency 75.907 ms 2024-09-20T23:51:15.313 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 podman[73530]: 2024-09-20 23:51:15.25216839 +0000 UTC m=+1.239803502 container remove 7ebad47c3a1351ee9ebe3729f5425567dfb9976b8722dde6c5bb1c8f02793e99 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-09-20T23:51:15.313 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.1.service: Deactivated successfully. 2024-09-20T23:51:15.313 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 systemd[1]: Stopped Ceph osd.1 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:51:15.313 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.1.service: Consumed 3min 14.553s CPU time. 2024-09-20T23:51:15.564 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:15 smithi137 ceph-mon[65321]: pgmap v91: 97 pgs: 11 active+undersized, 4 stale+active+clean, 17 active+undersized+degraded, 65 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.6 MiB/s wr, 517 op/s; 122/1617 objects degraded (7.545%) 2024-09-20T23:51:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:15 smithi144 ceph-mon[51829]: pgmap v91: 97 pgs: 11 active+undersized, 4 stale+active+clean, 17 active+undersized+degraded, 65 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 4.6 MiB/s wr, 517 op/s; 122/1617 objects degraded (7.545%) 2024-09-20T23:51:15.832 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 systemd[1]: Starting Ceph osd.1 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:51:16.109 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:15 smithi137 podman[73626]: 2024-09-20 23:51:15.824820133 +0000 UTC m=+0.016365661 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:51:16.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 187745 10.39 MB/sec execute 300 sec latency 87.543 ms 2024-09-20T23:51:16.368 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 podman[73626]: 2024-09-20 23:51:16.220740795 +0000 UTC m=+0.412286321 container create 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70) 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 podman[73626]: 2024-09-20 23:51:16.641774298 +0000 UTC m=+0.833319823 container init 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 podman[73626]: 2024-09-20 23:51:16.6529852 +0000 UTC m=+0.844530721 container start 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 bash[73626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 podman[73626]: 2024-09-20 23:51:16.830896253 +0000 UTC m=+1.022441784 container attach 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:51:16.871 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:16 smithi137 bash[73626]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:51:17.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 188108 10.40 MB/sec execute 301 sec latency 53.984 ms 2024-09-20T23:51:17.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:17 smithi137 ceph-mon[65321]: pgmap v92: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 555 op/s; 168/1593 objects degraded (10.546%) 2024-09-20T23:51:17.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:17 smithi144 ceph-mon[51829]: pgmap v92: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.4 MiB/s wr, 555 op/s; 168/1593 objects degraded (10.546%) 2024-09-20T23:51:18.168 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T23:51:18.168 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T23:51:18.168 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T23:51:18.168 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-1 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T23:51:18.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 188407 10.40 MB/sec execute 302 sec latency 78.354 ms 2024-09-20T23:51:18.424 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-1/block 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate[73641]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 bash[73626]: --> ceph-volume raw activate successful for osd ID: 1 2024-09-20T23:51:18.425 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:18 smithi137 podman[73626]: 2024-09-20 23:51:18.42366262 +0000 UTC m=+2.615208154 container died 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:51:19.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 188744 10.39 MB/sec execute 303 sec latency 55.369 ms 2024-09-20T23:51:19.443 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:19 smithi137 ceph-mon[65321]: pgmap v93: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.9 MiB/s wr, 290 op/s; 168/1593 objects degraded (10.546%) 2024-09-20T23:51:19.443 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:19.443 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:19 smithi144 ceph-mon[51829]: pgmap v93: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 2.9 MiB/s wr, 290 op/s; 168/1593 objects degraded (10.546%) 2024-09-20T23:51:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:19.703 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:19 smithi137 podman[73795]: 2024-09-20 23:51:19.456536359 +0000 UTC m=+1.025721200 container remove 34c5f3a552ad9a1d2b9ce6754731db6ab851f2b927ae82cc6718068060fd56cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-09-20T23:51:19.959 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:19 smithi137 podman[73884]: 2024-09-20 23:51:19.616179032 +0000 UTC m=+0.026850440 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:51:19.960 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:19 smithi137 podman[73884]: 2024-09-20 23:51:19.746941549 +0000 UTC m=+0.157612923 container create 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9e2ac70) 2024-09-20T23:51:20.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 189105 10.38 MB/sec execute 304 sec latency 40.304 ms 2024-09-20T23:51:20.373 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:20 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 184/1683 objects degraded (10.933%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:20.373 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:20 smithi137 podman[73884]: 2024-09-20 23:51:20.321681723 +0000 UTC m=+0.732353100 container init 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-20T23:51:20.373 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:20 smithi137 podman[73884]: 2024-09-20 23:51:20.328635183 +0000 UTC m=+0.739306570 container start 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240731) 2024-09-20T23:51:20.373 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:20 smithi137 ceph-osd[73903]: -- 172.21.15.137:0/1341673672 <== mon.0 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5633b54781e0 con 0x5633b4663c00 2024-09-20T23:51:20.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:20 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 184/1683 objects degraded (10.933%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:20.724 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:20 smithi137 bash[73884]: 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 2024-09-20T23:51:20.724 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:20 smithi137 systemd[1]: Started Ceph osd.1 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:51:21.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 189356 10.37 MB/sec execute 305 sec latency 51.644 ms 2024-09-20T23:51:21.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:21 smithi137 ceph-mon[65321]: pgmap v94: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 485 op/s; 184/1683 objects degraded (10.933%) 2024-09-20T23:51:21.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:21.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:21.764 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:51:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:21 smithi144 ceph-mon[51829]: pgmap v94: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.8 MiB/s wr, 485 op/s; 184/1683 objects degraded (10.933%) 2024-09-20T23:51:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:21.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:51:22.020 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:21 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:51:21.890+0000 7f6d07ed3640 -1 Falling back to public interface 2024-09-20T23:51:22.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 189776 10.37 MB/sec execute 306 sec latency 41.834 ms 2024-09-20T23:51:23.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 190211 10.36 MB/sec execute 307 sec latency 43.126 ms 2024-09-20T23:51:23.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:23 smithi137 ceph-mon[65321]: pgmap v95: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 446 op/s; 184/1692 objects degraded (10.875%) 2024-09-20T23:51:23.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:23 smithi144 ceph-mon[51829]: pgmap v95: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 339 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.6 MiB/s wr, 446 op/s; 184/1692 objects degraded (10.875%) 2024-09-20T23:51:24.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 190450 10.35 MB/sec execute 308 sec latency 67.910 ms 2024-09-20T23:51:25.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 190764 10.36 MB/sec execute 309 sec latency 62.535 ms 2024-09-20T23:51:25.788 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:25 smithi137 ceph-mon[65321]: pgmap v96: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.9 MiB/s wr, 468 op/s; 182/1605 objects degraded (11.340%) 2024-09-20T23:51:25.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:25 smithi144 ceph-mon[51829]: pgmap v96: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 334 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 4.9 MiB/s wr, 468 op/s; 182/1605 objects degraded (11.340%) 2024-09-20T23:51:26.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 191113 10.35 MB/sec execute 310 sec latency 52.664 ms 2024-09-20T23:51:26.818 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:26 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 182/1605 objects degraded (11.340%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:26.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:26 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 182/1605 objects degraded (11.340%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:27.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 191432 10.36 MB/sec execute 311 sec latency 54.518 ms 2024-09-20T23:51:27.849 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:27 smithi137 ceph-mon[65321]: pgmap v97: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 438 op/s; 181/1587 objects degraded (11.405%) 2024-09-20T23:51:27.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:27 smithi144 ceph-mon[51829]: pgmap v97: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.7 MiB/s wr, 438 op/s; 181/1587 objects degraded (11.405%) 2024-09-20T23:51:28.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 191663 10.35 MB/sec execute 312 sec latency 49.811 ms 2024-09-20T23:51:29.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 191856 10.34 MB/sec execute 313 sec latency 57.160 ms 2024-09-20T23:51:29.631 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-mon[65321]: pgmap v98: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 412 op/s; 181/1587 objects degraded (11.405%) 2024-09-20T23:51:29.632 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.632 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.632 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.632 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.883 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:29 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:51:29.827+0000 7f6d07ed3640 -1 osd.1 0 read_superblock omap replica is missing. 2024-09-20T23:51:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:29 smithi144 ceph-mon[51829]: pgmap v98: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 330 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.1 MiB/s wr, 412 op/s; 181/1587 objects degraded (11.405%) 2024-09-20T23:51:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:30.248 INFO:teuthology.orchestra.run.smithi137.stdout: 5 192084 10.32 MB/sec execute 314 sec latency 61.514 ms 2024-09-20T23:51:31.172 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:30 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:51:30.993+0000 7f6d07ed3640 -1 osd.1 86 log_to_monitors true 2024-09-20T23:51:31.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 192411 10.30 MB/sec execute 315 sec latency 79.347 ms 2024-09-20T23:51:31.708 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:31 smithi137 ceph-mon[65321]: pgmap v99: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 178/1590 objects degraded (11.195%) 2024-09-20T23:51:31.708 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:31 smithi137 ceph-mon[65321]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T23:51:31.708 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:31 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 178/1590 objects degraded (11.195%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:31 smithi144 ceph-mon[51829]: pgmap v99: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 335 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.4 KiB/s rd, 5.8 MiB/s wr, 571 op/s; 178/1590 objects degraded (11.195%) 2024-09-20T23:51:31.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:31 smithi144 ceph-mon[51829]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-20T23:51:31.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:31 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 178/1590 objects degraded (11.195%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:32.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 192689 10.29 MB/sec execute 316 sec latency 172.278 ms 2024-09-20T23:51:32.252 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:51:32 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:51:32.162+0000 7f6cfbc76640 -1 osd.1 86 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:51:32.754 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:32 smithi137 ceph-mon[65321]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T23:51:32.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:32 smithi137 ceph-mon[65321]: osdmap e89: 8 total, 7 up, 8 in 2024-09-20T23:51:32.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:32 smithi137 ceph-mon[65321]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:51:32.755 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:32 smithi137 ceph-mon[65321]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' 2024-09-20T23:51:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:32 smithi144 ceph-mon[51829]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-20T23:51:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:32 smithi144 ceph-mon[51829]: osdmap e89: 8 total, 7 up, 8 in 2024-09-20T23:51:32.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:32 smithi144 ceph-mon[51829]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:51:32.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:32 smithi144 ceph-mon[51829]: from='osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680]' entity='osd.1' 2024-09-20T23:51:33.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193051 10.29 MB/sec execute 317 sec latency 115.862 ms 2024-09-20T23:51:33.570 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: pgmap v101: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 467 op/s; 180/1617 objects degraded (11.132%) 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680] boot 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:51:33.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:33.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:33.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: pgmap v101: 97 pgs: 16 active+undersized, 23 active+undersized+degraded, 58 active+clean; 337 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 4.8 MiB/s wr, 467 op/s; 180/1617 objects degraded (11.132%) 2024-09-20T23:51:33.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:51:33.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: osd.1 [v2:172.21.15.137:6810/884130680,v1:172.21.15.137:6811/884130680] boot 2024-09-20T23:51:33.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: osdmap e90: 8 total, 8 up, 8 in 2024-09-20T23:51:33.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-20T23:51:33.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:33.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:34.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193051 10.26 MB/sec execute 318 sec latency 1115.934 ms 2024-09-20T23:51:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:51:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (4m) 5s ago 13m 25.5M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:51:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (99s) 5s ago 13m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:51:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (94s) 89s ago 13m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:51:34.312 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (4m) 5s ago 13m 53.8M - dad864ee21e9 c491347d0b92 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (7m) 5s ago 7m 41.0M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (7m) 89s ago 7m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (4m) 5s ago 15m 500M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (3m) 89s ago 13m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (2m) 5s ago 15m 48.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (2m) 89s ago 13m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (7m) 5s ago 7m 271M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (7m) 89s ago 7m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (4m) 5s ago 13m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (4m) 89s ago 12m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (60s) 5s ago 12m 230M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (14s) 5s ago 11m 11.7M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (11m) 5s ago 11m 451M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (10m) 5s ago 10m 592M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (10m) 89s ago 10m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:51:34.313 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (10m) 89s ago 10m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:51:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (9m) 89s ago 9m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:51:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (9m) 89s ago 9m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:51:34.314 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (3m) 5s ago 12m 64.4M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:51:34.737 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:34.738 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8, 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 6 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:51:34.739 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:51:34.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:34 smithi137 ceph-mon[65321]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:34.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:34 smithi137 ceph-mon[65321]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:34.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:34 smithi137 ceph-mon[65321]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T23:51:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:34 smithi144 ceph-mon[51829]: from='client.34236 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:34 smithi144 ceph-mon[51829]: from='client.34240 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:34.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:34 smithi144 ceph-mon[51829]: osdmap e91: 8 total, 8 up, 8 in 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:51:35.085 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "8/23 daemons upgraded", 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:51:35.086 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:51:35.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193051 10.22 MB/sec execute 319 sec latency 2115.966 ms 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN Degraded data redundancy: 66/1617 objects degraded (4.082%), 11 pgs degraded 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 66/1617 objects degraded (4.082%), 11 pgs degraded 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.4 is active+undersized+degraded, acting [0,7] 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.6 is active+undersized+degraded, acting [6,4] 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.8 is activating+degraded, acting [7,0,1] 2024-09-20T23:51:35.521 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.9 is active+undersized+degraded, acting [7,3] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.a is active+undersized+degraded, acting [3,7] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.d is active+undersized+degraded, acting [4,3] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.15 is active+undersized+degraded, acting [5,0] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.6 is active+undersized+degraded, acting [0,4] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.8 is active+undersized+degraded, acting [3,7] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.18 is active+undersized+degraded, acting [3,0] 2024-09-20T23:51:35.522 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.19 is active+undersized+degraded, acting [3,4] 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: pgmap v104: 97 pgs: 4 activating, 1 activating+degraded, 9 remapped+peering, 9 active+undersized, 10 active+undersized+degraded, 64 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 609 op/s; 66/1617 objects degraded (4.082%) 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/4071916573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T23:51:35.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:35.774 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:35 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1569482910' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:51:35.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: pgmap v104: 97 pgs: 4 activating, 1 activating+degraded, 9 remapped+peering, 9 active+undersized, 10 active+undersized+degraded, 64 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.0 MiB/s wr, 609 op/s; 66/1617 objects degraded (4.082%) 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='client.34244 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/4071916573' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: osdmap e92: 8 total, 8 up, 8 in 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:51:35.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:51:35.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:35.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:35 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/1569482910' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:51:36.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193180 10.20 MB/sec execute 320 sec latency 2676.242 ms 2024-09-20T23:51:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:36 smithi137 ceph-mon[65321]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:36 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:36 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:51:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:36 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 66/1617 objects degraded (4.082%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:36.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:36 smithi137 ceph-mon[65321]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T23:51:36.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:36 smithi144 ceph-mon[51829]: from='client.34252 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:51:36.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:36 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:36.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:36 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:51:36.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:36 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 66/1617 objects degraded (4.082%), 11 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:36.946 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:36 smithi144 ceph-mon[51829]: osdmap e93: 8 total, 8 up, 8 in 2024-09-20T23:51:37.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193203 10.17 MB/sec execute 321 sec latency 1035.563 ms 2024-09-20T23:51:37.721 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:37 smithi137 ceph-mon[65321]: pgmap v107: 97 pgs: 1 activating+undersized+degraded+remapped, 4 activating, 1 activating+degraded, 12 remapped+peering, 5 active+undersized, 6 active+undersized+degraded, 68 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 459 op/s; 44/1605 objects degraded (2.741%) 2024-09-20T23:51:37.722 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:37 smithi137 ceph-mon[65321]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T23:51:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:37 smithi144 ceph-mon[51829]: pgmap v107: 97 pgs: 1 activating+undersized+degraded+remapped, 4 activating, 1 activating+degraded, 12 remapped+peering, 5 active+undersized, 6 active+undersized+degraded, 68 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.9 MiB/s wr, 459 op/s; 44/1605 objects degraded (2.741%) 2024-09-20T23:51:37.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:37 smithi144 ceph-mon[51829]: osdmap e94: 8 total, 8 up, 8 in 2024-09-20T23:51:38.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193395 10.16 MB/sec execute 322 sec latency 823.193 ms 2024-09-20T23:51:39.223 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:39 smithi137 ceph-mon[65321]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T23:51:39.224 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:39 smithi137 ceph-mon[65321]: pgmap v110: 97 pgs: 1 activating+undersized+degraded+remapped, 4 activating, 1 activating+degraded, 12 remapped+peering, 79 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1004 KiB/s wr, 57 op/s; 20/1605 objects degraded (1.246%); 22 B/s, 49 keys/s, 0 objects/s recovering 2024-09-20T23:51:39.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 193803 10.14 MB/sec execute 323 sec latency 405.344 ms 2024-09-20T23:51:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:39 smithi144 ceph-mon[51829]: osdmap e95: 8 total, 8 up, 8 in 2024-09-20T23:51:39.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:39 smithi144 ceph-mon[51829]: pgmap v110: 97 pgs: 1 activating+undersized+degraded+remapped, 4 activating, 1 activating+degraded, 12 remapped+peering, 79 active+clean; 334 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 511 B/s rd, 1004 KiB/s wr, 57 op/s; 20/1605 objects degraded (1.246%); 22 B/s, 49 keys/s, 0 objects/s recovering 2024-09-20T23:51:40.239 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:40 smithi137 ceph-mon[65321]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T23:51:40.246 INFO:teuthology.orchestra.run.smithi137.stdout: 5 194054 10.14 MB/sec execute 324 sec latency 915.618 ms 2024-09-20T23:51:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:40 smithi144 ceph-mon[51829]: osdmap e96: 8 total, 8 up, 8 in 2024-09-20T23:51:41.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 194408 10.14 MB/sec execute 325 sec latency 643.663 ms 2024-09-20T23:51:41.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:41 smithi137 ceph-mon[65321]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T23:51:41.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:41 smithi137 ceph-mon[65321]: pgmap v113: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 88 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 232 op/s; 85/1542 objects degraded (5.512%); 1.9 MiB/s, 49 keys/s, 28 objects/s recovering 2024-09-20T23:51:41.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:41 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 20/1605 objects degraded (1.246%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:41.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:41 smithi137 ceph-mon[65321]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T23:51:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:41 smithi144 ceph-mon[51829]: osdmap e97: 8 total, 8 up, 8 in 2024-09-20T23:51:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:41 smithi144 ceph-mon[51829]: pgmap v113: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 activating+undersized+degraded+remapped, 1 peering, 2 remapped+peering, 88 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 1.9 MiB/s wr, 232 op/s; 85/1542 objects degraded (5.512%); 1.9 MiB/s, 49 keys/s, 28 objects/s recovering 2024-09-20T23:51:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:41 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 20/1605 objects degraded (1.246%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:41.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:41 smithi144 ceph-mon[51829]: osdmap e98: 8 total, 8 up, 8 in 2024-09-20T23:51:42.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 194719 10.14 MB/sec execute 326 sec latency 239.560 ms 2024-09-20T23:51:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 195045 10.15 MB/sec execute 327 sec latency 83.849 ms 2024-09-20T23:51:43.315 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:43 smithi137 ceph-mon[65321]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T23:51:43.316 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:43 smithi137 ceph-mon[65321]: pgmap v116: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 peering, 92 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 276 op/s; 63/1539 objects degraded (4.094%); 2.0 MiB/s, 46 objects/s recovering 2024-09-20T23:51:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:43 smithi144 ceph-mon[51829]: osdmap e99: 8 total, 8 up, 8 in 2024-09-20T23:51:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:43 smithi144 ceph-mon[51829]: pgmap v116: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 1 peering, 92 active+clean; 330 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.7 MiB/s wr, 276 op/s; 63/1539 objects degraded (4.094%); 2.0 MiB/s, 46 objects/s recovering 2024-09-20T23:51:44.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 195281 10.14 MB/sec execute 328 sec latency 473.506 ms 2024-09-20T23:51:44.374 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:44 smithi137 ceph-mon[65321]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T23:51:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:44 smithi144 ceph-mon[51829]: osdmap e100: 8 total, 8 up, 8 in 2024-09-20T23:51:45.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 195623 10.15 MB/sec execute 329 sec latency 47.566 ms 2024-09-20T23:51:45.393 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:45 smithi137 ceph-mon[65321]: pgmap v119: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 483 op/s; 45/1572 objects degraded (2.863%); 64 KiB/s, 23 objects/s recovering 2024-09-20T23:51:45.394 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:45 smithi137 ceph-mon[65321]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T23:51:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:45 smithi144 ceph-mon[51829]: pgmap v119: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 95 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.5 KiB/s rd, 4.2 MiB/s wr, 483 op/s; 45/1572 objects degraded (2.863%); 64 KiB/s, 23 objects/s recovering 2024-09-20T23:51:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:45 smithi144 ceph-mon[51829]: osdmap e101: 8 total, 8 up, 8 in 2024-09-20T23:51:46.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 196119 10.14 MB/sec execute 330 sec latency 39.698 ms 2024-09-20T23:51:46.408 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:46 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 45/1572 objects degraded (2.863%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:46 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 45/1572 objects degraded (2.863%), 2 pgs degraded (PG_DEGRADED) 2024-09-20T23:51:47.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 196676 10.14 MB/sec execute 331 sec latency 42.842 ms 2024-09-20T23:51:47.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:47 smithi137 ceph-mon[65321]: pgmap v120: 97 pgs: 1 active, 96 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.9 MiB/s wr, 427 op/s; 0 B/s, 23 objects/s recovering 2024-09-20T23:51:47.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:47 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1572 objects degraded (2.863%), 2 pgs degraded) 2024-09-20T23:51:47.419 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:47 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:51:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:47 smithi144 ceph-mon[51829]: pgmap v120: 97 pgs: 1 active, 96 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.9 MiB/s wr, 427 op/s; 0 B/s, 23 objects/s recovering 2024-09-20T23:51:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:47 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 45/1572 objects degraded (2.863%), 2 pgs degraded) 2024-09-20T23:51:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:47 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:51:48.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 197187 10.16 MB/sec execute 332 sec latency 40.633 ms 2024-09-20T23:51:49.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 197683 10.16 MB/sec execute 333 sec latency 54.248 ms 2024-09-20T23:51:49.444 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:49 smithi137 ceph-mon[65321]: pgmap v121: 97 pgs: 1 active, 96 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 479 op/s; 0 B/s, 10 objects/s recovering 2024-09-20T23:51:49.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:49.445 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:49 smithi144 ceph-mon[51829]: pgmap v121: 97 pgs: 1 active, 96 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 4.6 MiB/s wr, 479 op/s; 0 B/s, 10 objects/s recovering 2024-09-20T23:51:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:51:50.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 198133 10.17 MB/sec execute 334 sec latency 30.335 ms 2024-09-20T23:51:51.247 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:51 smithi137 ceph-mon[65321]: pgmap v122: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 515 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T23:51:51.247 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:51 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:51.247 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:51 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:51.247 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:51 smithi137 ceph-mon[65321]: Upgrade: osd.2 is safe to restart 2024-09-20T23:51:51.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 198553 10.19 MB/sec execute 335 sec latency 38.974 ms 2024-09-20T23:51:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:51 smithi144 ceph-mon[51829]: pgmap v122: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 9.7 KiB/s rd, 5.6 MiB/s wr, 515 op/s; 0 B/s, 7 objects/s recovering 2024-09-20T23:51:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:51 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:51 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["2"], "max": 16}]: dispatch 2024-09-20T23:51:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:51 smithi144 ceph-mon[51829]: Upgrade: osd.2 is safe to restart 2024-09-20T23:51:52.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 198919 10.19 MB/sec execute 336 sec latency 39.755 ms 2024-09-20T23:51:53.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 199287 10.20 MB/sec execute 337 sec latency 44.591 ms 2024-09-20T23:51:53.263 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:53 smithi137 ceph-mon[65321]: pgmap v123: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.3 MiB/s wr, 661 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T23:51:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:53 smithi144 ceph-mon[51829]: pgmap v123: 97 pgs: 97 active+clean; 341 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 6.3 MiB/s wr, 661 op/s; 0 B/s, 6 objects/s recovering 2024-09-20T23:51:54.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 199623 10.18 MB/sec execute 338 sec latency 38.348 ms 2024-09-20T23:51:54.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:54 smithi137 ceph-mon[65321]: Upgrade: Updating osd.2 2024-09-20T23:51:54.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:54.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:51:54.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:54.532 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:54 smithi137 ceph-mon[65321]: Deploying daemon osd.2 on smithi137 2024-09-20T23:51:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:54 smithi144 ceph-mon[51829]: Upgrade: Updating osd.2 2024-09-20T23:51:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-20T23:51:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:51:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:54 smithi144 ceph-mon[51829]: Deploying daemon osd.2 on smithi137 2024-09-20T23:51:55.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 199966 10.17 MB/sec execute 339 sec latency 45.644 ms 2024-09-20T23:51:55.557 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:55 smithi137 ceph-mon[65321]: pgmap v124: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.5 MiB/s wr, 637 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T23:51:55.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:55 smithi144 ceph-mon[51829]: pgmap v124: 97 pgs: 97 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.8 KiB/s rd, 6.5 MiB/s wr, 637 op/s; 0 B/s, 4 objects/s recovering 2024-09-20T23:51:56.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 200256 10.16 MB/sec execute 340 sec latency 84.004 ms 2024-09-20T23:51:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: 5 200557 10.15 MB/sec execute 341 sec latency 65.701 ms 2024-09-20T23:51:57.612 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:57 smithi137 ceph-mon[65321]: pgmap v125: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 557 op/s; 0 B/s, 3 objects/s recovering 2024-09-20T23:51:57.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:57 smithi144 ceph-mon[51829]: pgmap v125: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.1 MiB/s wr, 557 op/s; 0 B/s, 3 objects/s recovering 2024-09-20T23:51:58.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 200884 10.15 MB/sec execute 342 sec latency 52.097 ms 2024-09-20T23:51:58.894 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:51:58 smithi137 systemd[1]: Stopping Ceph osd.2 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:51:59.144 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[41818]: 2024-09-20T23:51:59.066+0000 7fe4d3cc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:51:59.145 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[41818]: 2024-09-20T23:51:59.066+0000 7fe4d3cc3700 -1 osd.2 101 *** Got signal Terminated *** 2024-09-20T23:51:59.145 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[41818]: 2024-09-20T23:51:59.066+0000 7fe4d3cc3700 -1 osd.2 101 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:51:59.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 201208 10.14 MB/sec execute 343 sec latency 65.971 ms 2024-09-20T23:51:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-mon[65321]: pgmap v126: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 537 op/s 2024-09-20T23:51:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:59.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:51:59 smithi137 ceph-mon[65321]: osd.2 marked itself down and dead 2024-09-20T23:51:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:59 smithi144 ceph-mon[51829]: pgmap v126: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 5.5 MiB/s wr, 537 op/s 2024-09-20T23:51:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:51:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:51:59 smithi144 ceph-mon[51829]: osd.2 marked itself down and dead 2024-09-20T23:52:00.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 201379 10.12 MB/sec execute 344 sec latency 880.104 ms 2024-09-20T23:52:00.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:00 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:52:00.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:00 smithi137 ceph-mon[65321]: osdmap e102: 8 total, 7 up, 8 in 2024-09-20T23:52:00.286 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:00 smithi137 podman[76006]: 2024-09-20 23:52:00.011318298 +0000 UTC m=+1.064787790 container died 5b535ad411d21ca8d0fde82bf6988905e6dd934c7d5b1e7048f2b629fe4f2184 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, version=8, name=centos-stream, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, RELEASE=HEAD, ceph=True, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-09-20T23:52:00.536 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:00 smithi137 podman[76006]: 2024-09-20 23:52:00.401719834 +0000 UTC m=+1.455189334 container cleanup 5b535ad411d21ca8d0fde82bf6988905e6dd934c7d5b1e7048f2b629fe4f2184 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, GIT_CLEAN=True, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, name=centos-stream, architecture=x86_64, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T23:52:00.537 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:00 smithi137 bash[76006]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2 2024-09-20T23:52:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:00 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:52:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:00 smithi144 ceph-mon[51829]: osdmap e102: 8 total, 7 up, 8 in 2024-09-20T23:52:00.735 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:00 smithi137 podman[76053]: 2024-09-20 23:52:00.730783795 +0000 UTC m=+0.722888930 container remove 5b535ad411d21ca8d0fde82bf6988905e6dd934c7d5b1e7048f2b629fe4f2184 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, io.openshift.expose-services=, distribution-scope=public, name=centos-stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, vcs-type=git, io.k8s.display-name=CentOS Stream 8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T23:52:01.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 201934 10.15 MB/sec execute 345 sec latency 1020.058 ms 2024-09-20T23:52:01.474 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:01 smithi137 podman[76136]: 2024-09-20 23:52:01.307330698 +0000 UTC m=+0.026778327 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:01.475 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:01 smithi137 ceph-mon[65321]: pgmap v128: 97 pgs: 5 stale+active+clean, 92 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.9 MiB/s wr, 650 op/s 2024-09-20T23:52:01.475 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:01 smithi137 ceph-mon[65321]: osdmap e103: 8 total, 7 up, 8 in 2024-09-20T23:52:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:01 smithi144 ceph-mon[51829]: pgmap v128: 97 pgs: 5 stale+active+clean, 92 active+clean; 333 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 5.9 MiB/s wr, 650 op/s 2024-09-20T23:52:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:01 smithi144 ceph-mon[51829]: osdmap e103: 8 total, 7 up, 8 in 2024-09-20T23:52:01.978 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:01 smithi137 podman[76136]: 2024-09-20 23:52:01.847645614 +0000 UTC m=+0.567093240 container create 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70) 2024-09-20T23:52:02.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 202678 10.19 MB/sec execute 346 sec latency 185.691 ms 2024-09-20T23:52:02.480 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:02 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 37/1605 objects degraded (2.305%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:02.481 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:02 smithi137 podman[76136]: 2024-09-20 23:52:02.342803621 +0000 UTC m=+1.062251273 container init 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0) 2024-09-20T23:52:02.481 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:02 smithi137 podman[76136]: 2024-09-20 23:52:02.349000412 +0000 UTC m=+1.068448041 container start 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70) 2024-09-20T23:52:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:02 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 37/1605 objects degraded (2.305%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:02.731 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:02 smithi137 podman[76136]: 2024-09-20 23:52:02.593434527 +0000 UTC m=+1.312890107 container attach 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T23:52:02.983 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:02 smithi137 podman[76136]: 2024-09-20 23:52:02.746567135 +0000 UTC m=+1.466014761 container died 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:52:03.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 203305 10.20 MB/sec execute 347 sec latency 30.416 ms 2024-09-20T23:52:03.485 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:03 smithi137 ceph-mon[65321]: pgmap v130: 97 pgs: 3 active+undersized, 2 peering, 3 stale+active+clean, 3 active+undersized+degraded, 86 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 345 op/s; 37/1605 objects degraded (2.305%) 2024-09-20T23:52:03.486 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:03.486 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:03 smithi144 ceph-mon[51829]: pgmap v130: 97 pgs: 3 active+undersized, 2 peering, 3 stale+active+clean, 3 active+undersized+degraded, 86 active+clean; 330 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 345 op/s; 37/1605 objects degraded (2.305%) 2024-09-20T23:52:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:03.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:04.004 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:03 smithi137 podman[76158]: 2024-09-20 23:52:03.835773993 +0000 UTC m=+1.078011157 container remove 09696c923d9e7ff28e42767123b066389652b0a38167660bdce63b71c385507a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2) 2024-09-20T23:52:04.004 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:03 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.2.service: Deactivated successfully. 2024-09-20T23:52:04.004 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:03 smithi137 systemd[1]: Stopped Ceph osd.2 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:52:04.004 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:03 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.2.service: Consumed 2min 24.775s CPU time. 2024-09-20T23:52:04.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 203842 10.21 MB/sec execute 348 sec latency 49.650 ms 2024-09-20T23:52:04.506 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:04 smithi137 systemd[1]: Starting Ceph osd.2 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:52:04.561 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:04 smithi137 podman[76255]: 2024-09-20 23:52:04.412273827 +0000 UTC m=+0.022846115 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:04.562 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:04 smithi137 podman[76255]: 2024-09-20 23:52:04.529021359 +0000 UTC m=+0.139593626 container create 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, ceph=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T23:52:05.009 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:04 smithi137 podman[76255]: 2024-09-20 23:52:04.908220654 +0000 UTC m=+0.518792937 container init 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True) 2024-09-20T23:52:05.010 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:04 smithi137 podman[76255]: 2024-09-20 23:52:04.915917882 +0000 UTC m=+0.526490145 container start 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70) 2024-09-20T23:52:05.010 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:05 smithi137 podman[76255]: 2024-09-20 23:52:05.008046736 +0000 UTC m=+0.618619004 container attach 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:52:05.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 204247 10.21 MB/sec execute 349 sec latency 59.431 ms 2024-09-20T23:52:05.262 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:05 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:05.263 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:05 smithi137 bash[76255]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:05.263 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:05 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:05.263 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:05 smithi137 bash[76255]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:05.513 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:05 smithi137 ceph-mon[65321]: pgmap v131: 97 pgs: 5 active+undersized, 2 peering, 2 stale+active+clean, 13 active+undersized+degraded, 75 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 6.0 MiB/s wr, 670 op/s; 129/1758 objects degraded (7.338%) 2024-09-20T23:52:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:05 smithi144 ceph-mon[51829]: pgmap v131: 97 pgs: 5 active+undersized, 2 peering, 2 stale+active+clean, 13 active+undersized+degraded, 75 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 6.0 MiB/s wr, 670 op/s; 129/1758 objects degraded (7.338%) 2024-09-20T23:52:06.088 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:52:06.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 204654 10.21 MB/sec execute 350 sec latency 45.168 ms 2024-09-20T23:52:06.270 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T23:52:06.271 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T23:52:06.271 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T23:52:06.271 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-2 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T23:52:06.524 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-2/block 2024-09-20T23:52:06.525 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T23:52:06.525 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-09-20T23:52:06.525 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate[76268]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T23:52:06.525 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 bash[76255]: --> ceph-volume raw activate successful for osd ID: 2 2024-09-20T23:52:06.525 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:06 smithi137 podman[76255]: 2024-09-20 23:52:06.514679376 +0000 UTC m=+2.125251638 container died 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T23:52:07.139 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:07.139 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (5m) 38s ago 14m 25.5M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:52:07.139 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (2m) 38s ago 14m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (2m) 2m ago 13m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (5m) 38s ago 14m 53.8M - dad864ee21e9 c491347d0b92 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (8m) 38s ago 8m 41.0M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (8m) 2m ago 8m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (4m) 38s ago 15m 500M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (3m) 2m ago 13m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (3m) 38s ago 15m 48.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (3m) 2m ago 13m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (8m) 38s ago 8m 271M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (7m) 2m ago 7m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (5m) 38s ago 14m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (4m) 2m ago 13m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (93s) 38s ago 12m 230M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:52:07.140 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (47s) 38s ago 12m 11.7M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (11m) 38s ago 11m 451M 1936M 17.2.0 e1d6a67b021e 5b535ad411d2 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (11m) 38s ago 11m 592M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (11m) 2m ago 11m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (10m) 2m ago 10m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (10m) 2m ago 10m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (9m) 2m ago 9m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:52:07.141 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (3m) 38s ago 13m 64.4M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:52:07.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 205335 10.23 MB/sec execute 351 sec latency 21.278 ms 2024-09-20T23:52:07.527 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:07 smithi137 ceph-mon[65321]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:07.528 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:07 smithi137 ceph-mon[65321]: pgmap v132: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 763 op/s; 169/1821 objects degraded (9.281%) 2024-09-20T23:52:07.528 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:07 smithi137 ceph-mon[65321]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:07 smithi144 ceph-mon[51829]: from='client.34260 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:07 smithi144 ceph-mon[51829]: pgmap v132: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 344 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.2 KiB/s rd, 7.5 MiB/s wr, 763 op/s; 169/1821 objects degraded (9.281%) 2024-09-20T23:52:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:07 smithi144 ceph-mon[51829]: from='client.34264 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:07.720 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:52:07.720 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:52:07.720 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-20T23:52:07.721 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 6 2024-09-20T23:52:07.722 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:52:07.722 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "8/23 daemons upgraded", 2024-09-20T23:52:08.248 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:52:08.249 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:52:08.249 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:52:08.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 205863 10.26 MB/sec execute 352 sec latency 36.983 ms 2024-09-20T23:52:08.287 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:08 smithi137 podman[76456]: 2024-09-20 23:52:08.131529963 +0000 UTC m=+1.592149973 container remove 2d432c438832b389d574a95676ef29a627918137ee49139eee5706493fb5474a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-activate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-20T23:52:08.538 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:08 smithi137 podman[76584]: 2024-09-20 23:52:08.344334734 +0000 UTC m=+0.034894276 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:08.539 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:08 smithi137 podman[76584]: 2024-09-20 23:52:08.536754075 +0000 UTC m=+0.227313607 container create 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:52:08.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:08 smithi137 ceph-mon[65321]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:08.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:08 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2316125481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:08.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:08 smithi144 ceph-mon[51829]: from='client.44215 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:08.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:08 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2316125481' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 167/1803 objects degraded (9.262%), 19 pgs degraded 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: osd.2 (root=default,host=smithi137) is down 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 167/1803 objects degraded (9.262%), 19 pgs degraded 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1 is active+undersized+degraded, acting [3,0] 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.3 is active+undersized+degraded, acting [5,7] 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.7 is active+undersized+degraded, acting [6,7] 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.c is active+undersized+degraded, acting [5,0] 2024-09-20T23:52:08.970 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.e is active+undersized+degraded, acting [3,7] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.10 is active+undersized+degraded, acting [1,0] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.16 is active+undersized+degraded, acting [5,6] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.17 is active+undersized+degraded, acting [6,5] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1c is active+undersized+degraded, acting [4,5] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.0 is active+undersized+degraded, acting [1,6] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.4 is active+undersized+degraded, acting [1,5] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.5 is active+undersized+degraded, acting [5,3] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.9 is active+undersized+degraded, acting [4,7] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.13 is active+undersized+degraded, acting [7,4] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1a is active+undersized+degraded, acting [4,1] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1e is active+undersized+degraded, acting [3,6] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1f is active+undersized+degraded, acting [0,5] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.14 is active+undersized+degraded, acting [3,7] 2024-09-20T23:52:08.971 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.19 is active+undersized+degraded, acting [3,0] 2024-09-20T23:52:09.050 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:08 smithi137 podman[76584]: 2024-09-20 23:52:08.93304153 +0000 UTC m=+0.623601080 container init 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T23:52:09.050 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:08 smithi137 podman[76584]: 2024-09-20 23:52:08.956283777 +0000 UTC m=+0.646843303 container start 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70) 2024-09-20T23:52:09.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 206313 10.27 MB/sec execute 353 sec latency 49.288 ms 2024-09-20T23:52:09.311 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:09 smithi137 bash[76584]: 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 2024-09-20T23:52:09.311 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:09 smithi137 systemd[1]: Started Ceph osd.2 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: pgmap v133: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.1 MiB/s wr, 462 op/s; 167/1803 objects degraded (9.262%) 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 167/1803 objects degraded (9.262%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2846592162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:09.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: pgmap v133: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 5.1 MiB/s wr, 462 op/s; 167/1803 objects degraded (9.262%) 2024-09-20T23:52:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: from='client.44223 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 167/1803 objects degraded (9.262%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:09.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2846592162' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:52:09.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:09.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:09.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:10.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 206671 10.26 MB/sec execute 354 sec latency 38.299 ms 2024-09-20T23:52:10.922 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:10 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:52:10.855+0000 7fab1c16c640 -1 Falling back to public interface 2024-09-20T23:52:11.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 207044 10.26 MB/sec execute 355 sec latency 50.991 ms 2024-09-20T23:52:11.609 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:11 smithi137 ceph-mon[65321]: pgmap v134: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.5 MiB/s wr, 722 op/s; 156/1683 objects degraded (9.269%) 2024-09-20T23:52:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:11 smithi144 ceph-mon[51829]: pgmap v134: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 7.5 MiB/s wr, 722 op/s; 156/1683 objects degraded (9.269%) 2024-09-20T23:52:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 207536 10.25 MB/sec execute 356 sec latency 83.800 ms 2024-09-20T23:52:13.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 207916 10.25 MB/sec execute 357 sec latency 38.622 ms 2024-09-20T23:52:13.645 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:13 smithi137 ceph-mon[65321]: pgmap v135: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.7 MiB/s wr, 628 op/s; 162/1722 objects degraded (9.408%) 2024-09-20T23:52:13.646 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:13 smithi144 ceph-mon[51829]: pgmap v135: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.4 KiB/s rd, 6.7 MiB/s wr, 628 op/s; 162/1722 objects degraded (9.408%) 2024-09-20T23:52:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:14.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 208217 10.26 MB/sec execute 358 sec latency 76.570 ms 2024-09-20T23:52:14.675 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:14 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 158/1674 objects degraded (9.438%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:14 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 158/1674 objects degraded (9.438%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:15.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 208560 10.25 MB/sec execute 359 sec latency 60.491 ms 2024-09-20T23:52:15.701 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:15 smithi137 ceph-mon[65321]: pgmap v136: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.1 MiB/s wr, 819 op/s; 158/1674 objects degraded (9.438%) 2024-09-20T23:52:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:15 smithi144 ceph-mon[51829]: pgmap v136: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 343 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 8.1 MiB/s wr, 819 op/s; 158/1674 objects degraded (9.438%) 2024-09-20T23:52:16.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 208833 10.24 MB/sec execute 360 sec latency 55.027 ms 2024-09-20T23:52:17.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 209037 10.22 MB/sec execute 361 sec latency 55.065 ms 2024-09-20T23:52:17.744 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:17 smithi137 ceph-mon[65321]: pgmap v137: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 609 op/s; 156/1629 objects degraded (9.576%) 2024-09-20T23:52:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:17.745 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:17 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:17.745 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:17 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:52:17.649+0000 7fab1c16c640 -1 osd.2 0 read_superblock omap replica is missing. 2024-09-20T23:52:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:17 smithi144 ceph-mon[51829]: pgmap v137: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 340 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 6.5 MiB/s wr, 609 op/s; 156/1629 objects degraded (9.576%) 2024-09-20T23:52:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:17 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:18.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 209294 10.23 MB/sec execute 362 sec latency 57.653 ms 2024-09-20T23:52:18.762 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:52:18.591+0000 7fab1c16c640 -1 osd.2 101 log_to_monitors true 2024-09-20T23:52:19.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 209570 10.23 MB/sec execute 363 sec latency 53.012 ms 2024-09-20T23:52:19.273 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:52:19.230+0000 7fab13f17640 -1 osd.2 101 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:52:19.273 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:19.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:19.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: pgmap v138: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 152/1614 objects degraded (9.418%) 2024-09-20T23:52:19.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:19.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: from='osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:52:19.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:19 smithi137 ceph-mon[65321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:52:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: pgmap v138: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 335 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.5 MiB/s wr, 548 op/s; 152/1614 objects degraded (9.418%) 2024-09-20T23:52:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:19.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: from='osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:52:19.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:19 smithi144 ceph-mon[51829]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-20T23:52:20.255 INFO:teuthology.orchestra.run.smithi137.stdout: 5 209856 10.22 MB/sec execute 364 sec latency 59.910 ms 2024-09-20T23:52:20.285 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:20 smithi137 ceph-mon[65321]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T23:52:20.285 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:20 smithi137 ceph-mon[65321]: osdmap e104: 8 total, 7 up, 8 in 2024-09-20T23:52:20.285 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:20 smithi137 ceph-mon[65321]: from='osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:52:20.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:20 smithi137 ceph-mon[65321]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:52:20.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:20 smithi144 ceph-mon[51829]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-20T23:52:20.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:20 smithi144 ceph-mon[51829]: osdmap e104: 8 total, 7 up, 8 in 2024-09-20T23:52:20.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:20 smithi144 ceph-mon[51829]: from='osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:52:20.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:20 smithi144 ceph-mon[51829]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:52:21.256 INFO:teuthology.orchestra.run.smithi137.stdout: 5 209992 10.19 MB/sec execute 365 sec latency 1059.943 ms 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075] boot 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: pgmap v141: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.3 MiB/s wr, 540 op/s; 150/1632 objects degraded (9.191%) 2024-09-20T23:52:21.436 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:21 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 152/1614 objects degraded (9.418%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: osd.2 [v2:172.21.15.137:6818/1072738075,v1:172.21.15.137:6819/1072738075] boot 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: osdmap e105: 8 total, 8 up, 8 in 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: pgmap v141: 97 pgs: 6 active+undersized, 19 active+undersized+degraded, 72 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.9 KiB/s rd, 5.3 MiB/s wr, 540 op/s; 150/1632 objects degraded (9.191%) 2024-09-20T23:52:21.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:21 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 152/1614 objects degraded (9.418%), 19 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:22.207 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:22 smithi137 ceph-mon[65321]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T23:52:22.208 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:22 smithi137 ceph-mon[65321]: osdmap e107: 8 total, 8 up, 8 in 2024-09-20T23:52:22.261 INFO:teuthology.orchestra.run.smithi137.stdout: 5 210163 10.18 MB/sec execute 366 sec latency 1793.079 ms 2024-09-20T23:52:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:22 smithi144 ceph-mon[51829]: osdmap e106: 8 total, 8 up, 8 in 2024-09-20T23:52:22.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:22 smithi144 ceph-mon[51829]: osdmap e107: 8 total, 8 up, 8 in 2024-09-20T23:52:23.233 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:23 smithi137 ceph-mon[65321]: pgmap v144: 97 pgs: 2 remapped+peering, 2 peering, 3 active+undersized, 16 active+undersized+degraded, 74 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.1 MiB/s wr, 589 op/s; 111/1626 objects degraded (6.827%) 2024-09-20T23:52:23.263 INFO:teuthology.orchestra.run.smithi137.stdout: 5 210428 10.17 MB/sec execute 367 sec latency 1820.391 ms 2024-09-20T23:52:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:23 smithi144 ceph-mon[51829]: pgmap v144: 97 pgs: 2 remapped+peering, 2 peering, 3 active+undersized, 16 active+undersized+degraded, 74 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.1 MiB/s wr, 589 op/s; 111/1626 objects degraded (6.827%) 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: osdmap e108: 8 total, 8 up, 8 in 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:24.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:24 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:52:24.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 210711 10.15 MB/sec execute 368 sec latency 36.693 ms 2024-09-20T23:52:24.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: osdmap e108: 8 total, 8 up, 8 in 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:24 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:52:25.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 211002 10.15 MB/sec execute 369 sec latency 456.241 ms 2024-09-20T23:52:25.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:25 smithi137 ceph-mon[65321]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T23:52:25.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:25 smithi137 ceph-mon[65321]: pgmap v147: 97 pgs: 2 active+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 86 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 325 op/s; 27/1596 objects degraded (1.692%); 2.0 MiB/s, 23 objects/s recovering 2024-09-20T23:52:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:25 smithi144 ceph-mon[51829]: osdmap e109: 8 total, 8 up, 8 in 2024-09-20T23:52:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:25 smithi144 ceph-mon[51829]: pgmap v147: 97 pgs: 2 active+undersized+remapped, 2 remapped+peering, 3 peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+degraded, 86 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 3.7 MiB/s wr, 325 op/s; 27/1596 objects degraded (1.692%); 2.0 MiB/s, 23 objects/s recovering 2024-09-20T23:52:26.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 211325 10.14 MB/sec execute 370 sec latency 67.695 ms 2024-09-20T23:52:26.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:26 smithi137 ceph-mon[65321]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T23:52:26.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:26 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 27/1596 objects degraded (1.692%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:26.299 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:26 smithi137 ceph-mon[65321]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T23:52:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:26 smithi144 ceph-mon[51829]: osdmap e110: 8 total, 8 up, 8 in 2024-09-20T23:52:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:26 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 27/1596 objects degraded (1.692%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:26 smithi144 ceph-mon[51829]: osdmap e111: 8 total, 8 up, 8 in 2024-09-20T23:52:27.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 211617 10.14 MB/sec execute 371 sec latency 225.285 ms 2024-09-20T23:52:27.307 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:27 smithi137 ceph-mon[65321]: pgmap v150: 97 pgs: 2 active+undersized+remapped, 2 remapped+peering, 2 peering, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 259 op/s; 14/1617 objects degraded (0.866%); 2.1 MiB/s, 36 objects/s recovering 2024-09-20T23:52:27.307 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:27 smithi137 ceph-mon[65321]: osdmap e112: 8 total, 8 up, 8 in 2024-09-20T23:52:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:27 smithi144 ceph-mon[51829]: pgmap v150: 97 pgs: 2 active+undersized+remapped, 2 remapped+peering, 2 peering, 1 active+recovering+undersized+degraded+remapped, 90 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 2.9 MiB/s wr, 259 op/s; 14/1617 objects degraded (0.866%); 2.1 MiB/s, 36 objects/s recovering 2024-09-20T23:52:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:27 smithi144 ceph-mon[51829]: osdmap e112: 8 total, 8 up, 8 in 2024-09-20T23:52:28.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 211859 10.14 MB/sec execute 372 sec latency 74.509 ms 2024-09-20T23:52:29.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 212139 10.13 MB/sec execute 373 sec latency 53.570 ms 2024-09-20T23:52:29.381 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:29 smithi137 ceph-mon[65321]: pgmap v152: 97 pgs: 2 active+undersized+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s wr, 246 op/s; 14/1611 objects degraded (0.869%); 233 KiB/s, 26 objects/s recovering 2024-09-20T23:52:29.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:29 smithi144 ceph-mon[51829]: pgmap v152: 97 pgs: 2 active+undersized+remapped, 3 peering, 1 active+recovering+undersized+degraded+remapped, 91 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 2.0 MiB/s wr, 246 op/s; 14/1611 objects degraded (0.869%); 233 KiB/s, 26 objects/s recovering 2024-09-20T23:52:30.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 212380 10.11 MB/sec execute 374 sec latency 61.554 ms 2024-09-20T23:52:30.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:30 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1611 objects degraded (0.869%), 1 pg degraded) 2024-09-20T23:52:30.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:30 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:52:30.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:30 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1611 objects degraded (0.869%), 1 pg degraded) 2024-09-20T23:52:30.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:30 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:52:31.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 212596 10.10 MB/sec execute 375 sec latency 64.157 ms 2024-09-20T23:52:31.468 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:31 smithi137 ceph-mon[65321]: pgmap v153: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 472 op/s; 156 KiB/s, 22 objects/s recovering 2024-09-20T23:52:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:31 smithi144 ceph-mon[51829]: pgmap v153: 97 pgs: 1 peering, 96 active+clean; 335 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 MiB/s wr, 472 op/s; 156 KiB/s, 22 objects/s recovering 2024-09-20T23:52:32.273 INFO:teuthology.orchestra.run.smithi137.stdout: 5 212961 10.11 MB/sec execute 376 sec latency 75.527 ms 2024-09-20T23:52:33.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 213248 10.10 MB/sec execute 377 sec latency 50.843 ms 2024-09-20T23:52:33.348 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:33 smithi137 ceph-mon[65321]: pgmap v154: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.3 MiB/s wr, 346 op/s; 134 KiB/s, 15 objects/s recovering 2024-09-20T23:52:33.348 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:33.348 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:33 smithi144 ceph-mon[51829]: pgmap v154: 97 pgs: 97 active+clean; 334 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 4.4 KiB/s rd, 3.3 MiB/s wr, 346 op/s; 134 KiB/s, 15 objects/s recovering 2024-09-20T23:52:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:34.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 213556 10.10 MB/sec execute 378 sec latency 60.832 ms 2024-09-20T23:52:35.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 213902 10.09 MB/sec execute 379 sec latency 67.352 ms 2024-09-20T23:52:35.391 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:35 smithi137 ceph-mon[65321]: pgmap v155: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 509 op/s; 106 KiB/s, 10 objects/s recovering 2024-09-20T23:52:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:35 smithi144 ceph-mon[51829]: pgmap v155: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.7 KiB/s rd, 4.7 MiB/s wr, 509 op/s; 106 KiB/s, 10 objects/s recovering 2024-09-20T23:52:36.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 214222 10.08 MB/sec execute 380 sec latency 43.431 ms 2024-09-20T23:52:37.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 214537 10.08 MB/sec execute 381 sec latency 43.939 ms 2024-09-20T23:52:37.425 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:37 smithi137 ceph-mon[65321]: pgmap v156: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 437 op/s; 85 KiB/s, 8 objects/s recovering 2024-09-20T23:52:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:37 smithi144 ceph-mon[51829]: pgmap v156: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.3 MiB/s wr, 437 op/s; 85 KiB/s, 8 objects/s recovering 2024-09-20T23:52:38.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 214956 10.08 MB/sec execute 382 sec latency 45.565 ms 2024-09-20T23:52:39.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 215285 10.08 MB/sec execute 383 sec latency 98.204 ms 2024-09-20T23:52:39.386 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:39 smithi137 ceph-mon[65321]: pgmap v157: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 380 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T23:52:39.387 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:39.387 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:39.387 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:39 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:39.387 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:39 smithi137 ceph-mon[65321]: Upgrade: osd.3 is safe to restart 2024-09-20T23:52:39.534 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:52:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:39 smithi144 ceph-mon[51829]: pgmap v157: 97 pgs: 97 active+clean; 337 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 3.9 MiB/s wr, 380 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T23:52:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:39 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["3"], "max": 16}]: dispatch 2024-09-20T23:52:39.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:39 smithi144 ceph-mon[51829]: Upgrade: osd.3 is safe to restart 2024-09-20T23:52:40.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 215675 10.09 MB/sec execute 384 sec latency 57.411 ms 2024-09-20T23:52:40.495 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:40 smithi137 ceph-mon[65321]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:40.669 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:52:40.669 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (6m) 23s ago 15m 25.5M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (2m) 23s ago 14m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (2m) 2m ago 14m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (5m) 23s ago 14m 54.5M - dad864ee21e9 c491347d0b92 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (8m) 23s ago 8m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (8m) 2m ago 8m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (5m) 23s ago 16m 502M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (4m) 2m ago 14m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (4m) 23s ago 16m 50.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (3m) 2m ago 14m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (8m) 23s ago 8m 280M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (8m) 2m ago 8m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (5m) 23s ago 14m 16.4M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (5m) 2m ago 14m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:52:40.670 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (2m) 23s ago 13m 289M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (81s) 23s ago 12m 225M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (32s) 23s ago 12m 14.4M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (12m) 23s ago 12m 376M 1936M 17.2.0 e1d6a67b021e dcef0bd0fb3f 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (11m) 2m ago 11m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (11m) 2m ago 11m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (10m) 2m ago 10m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (10m) 2m ago 10m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:52:40.671 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (4m) 23s ago 13m 64.4M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:52:40.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:40 smithi144 ceph-mon[51829]: from='client.34288 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:41.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 216045 10.08 MB/sec execute 385 sec latency 39.106 ms 2024-09-20T23:52:41.313 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 3 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:52:41.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:52:41.315 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:52:41.315 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 7, 2024-09-20T23:52:41.315 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 7 2024-09-20T23:52:41.315 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:52:41.315 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:52:41.516 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:41.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: pgmap v158: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.1 MiB/s wr, 498 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T23:52:41.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:41.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:41.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:52:41.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:52:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: from='client.34292 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: pgmap v158: 97 pgs: 97 active+clean; 336 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.5 KiB/s rd, 5.1 MiB/s wr, 498 op/s; 0 B/s, 2 objects/s recovering 2024-09-20T23:52:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: from='client.34296 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-20T23:52:41.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:52:41.938 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "9/23 daemons upgraded", 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:52:41.939 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:52:42.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 216299 10.07 MB/sec execute 386 sec latency 36.778 ms 2024-09-20T23:52:42.528 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:42 smithi137 ceph-mon[65321]: Upgrade: Updating osd.3 2024-09-20T23:52:42.529 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:42 smithi137 ceph-mon[65321]: Deploying daemon osd.3 on smithi137 2024-09-20T23:52:42.529 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:42 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1118651449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:42.529 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:42 smithi137 ceph-mon[65321]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:42.586 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:52:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:42 smithi144 ceph-mon[51829]: Upgrade: Updating osd.3 2024-09-20T23:52:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:42 smithi144 ceph-mon[51829]: Deploying daemon osd.3 on smithi137 2024-09-20T23:52:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:42 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/1118651449' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:52:42.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:42 smithi144 ceph-mon[51829]: from='client.44245 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:52:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 216619 10.07 MB/sec execute 387 sec latency 53.579 ms 2024-09-20T23:52:43.538 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:43 smithi137 ceph-mon[65321]: pgmap v159: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 457 op/s 2024-09-20T23:52:43.539 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:43 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3687835600' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:52:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:43 smithi144 ceph-mon[51829]: pgmap v159: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.5 MiB/s wr, 457 op/s 2024-09-20T23:52:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:43 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3687835600' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:52:44.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 216968 10.08 MB/sec execute 388 sec latency 39.444 ms 2024-09-20T23:52:45.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 217327 10.07 MB/sec execute 389 sec latency 49.592 ms 2024-09-20T23:52:45.325 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:45 smithi137 ceph-mon[65321]: pgmap v160: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 477 op/s 2024-09-20T23:52:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:45 smithi144 ceph-mon[51829]: pgmap v160: 97 pgs: 97 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.6 KiB/s rd, 5.0 MiB/s wr, 477 op/s 2024-09-20T23:52:46.088 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:45 smithi137 systemd[1]: Stopping Ceph osd.3 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:52:46.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 217605 10.06 MB/sec execute 390 sec latency 44.976 ms 2024-09-20T23:52:46.339 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:46 smithi137 ceph-mon[65321]: osd.3 marked itself down and dead 2024-09-20T23:52:46.340 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:46 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[45060]: 2024-09-20T23:52:46.265+0000 7f5f963c5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:52:46.340 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:46 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[45060]: 2024-09-20T23:52:46.265+0000 7f5f963c5700 -1 osd.3 112 *** Got signal Terminated *** 2024-09-20T23:52:46.340 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:46 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[45060]: 2024-09-20T23:52:46.265+0000 7f5f963c5700 -1 osd.3 112 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:52:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:46 smithi144 ceph-mon[51829]: osd.3 marked itself down and dead 2024-09-20T23:52:47.097 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:47 smithi137 podman[78770]: 2024-09-20 23:52:47.001543012 +0000 UTC m=+0.934712849 container died dcef0bd0fb3f151c940edb877eea267099e752acd74befa295a1eb90731a4d1b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, release=754, GIT_CLEAN=True, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , RELEASE=HEAD, ceph=True, architecture=x86_64, version=8) 2024-09-20T23:52:47.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 217740 10.03 MB/sec execute 391 sec latency 953.713 ms 2024-09-20T23:52:47.564 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:47 smithi137 podman[78770]: 2024-09-20 23:52:47.557647593 +0000 UTC m=+1.490817416 container cleanup dcef0bd0fb3f151c940edb877eea267099e752acd74befa295a1eb90731a4d1b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, architecture=x86_64, version=8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., release=754, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, name=centos-stream, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public) 2024-09-20T23:52:47.565 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:47 smithi137 bash[78770]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3 2024-09-20T23:52:47.565 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:47 smithi137 ceph-mon[65321]: pgmap v161: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 437 op/s 2024-09-20T23:52:47.565 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:47 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:52:47.565 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:47 smithi137 ceph-mon[65321]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T23:52:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:47 smithi144 ceph-mon[51829]: pgmap v161: 97 pgs: 97 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.1 KiB/s rd, 4.7 MiB/s wr, 437 op/s 2024-09-20T23:52:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:47 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:52:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:47 smithi144 ceph-mon[51829]: osdmap e113: 8 total, 7 up, 8 in 2024-09-20T23:52:48.082 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:47 smithi137 podman[78786]: 2024-09-20 23:52:47.909665367 +0000 UTC m=+0.908472723 container remove dcef0bd0fb3f151c940edb877eea267099e752acd74befa295a1eb90731a4d1b (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, name=centos-stream, build-date=2022-05-03T08:36:31.336870, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, io.buildah.version=1.19.8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-09-20T23:52:48.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 217943 10.02 MB/sec execute 392 sec latency 1041.210 ms 2024-09-20T23:52:48.334 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:48 smithi137 podman[78872]: 2024-09-20 23:52:48.195601239 +0000 UTC m=+0.021522485 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:48.334 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:48 smithi137 podman[78872]: 2024-09-20 23:52:48.327597695 +0000 UTC m=+0.153518928 container create bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-20T23:52:48.589 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:48 smithi137 ceph-mon[65321]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T23:52:48.590 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:48.590 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:48 smithi144 ceph-mon[51829]: osdmap e114: 8 total, 7 up, 8 in 2024-09-20T23:52:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:52:49.141 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:48 smithi137 podman[78872]: 2024-09-20 23:52:48.974809732 +0000 UTC m=+0.800730992 container init bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T23:52:49.141 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:48 smithi137 podman[78872]: 2024-09-20 23:52:48.981791272 +0000 UTC m=+0.807712512 container start bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70) 2024-09-20T23:52:49.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 218198 10.02 MB/sec execute 393 sec latency 69.138 ms 2024-09-20T23:52:49.405 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:49 smithi137 ceph-mon[65321]: pgmap v164: 97 pgs: 9 peering, 13 stale+active+clean, 75 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 402 op/s 2024-09-20T23:52:49.406 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:49 smithi137 ceph-mon[65321]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T23:52:49.406 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:49 smithi137 podman[78872]: 2024-09-20 23:52:49.157992056 +0000 UTC m=+0.983913295 container attach bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS) 2024-09-20T23:52:49.406 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:49 smithi137 podman[78872]: 2024-09-20 23:52:49.267930856 +0000 UTC m=+1.093852106 container died bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-20T23:52:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:49 smithi144 ceph-mon[51829]: pgmap v164: 97 pgs: 9 peering, 13 stale+active+clean, 75 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 4.1 KiB/s rd, 4.1 MiB/s wr, 402 op/s 2024-09-20T23:52:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:49 smithi144 ceph-mon[51829]: Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2024-09-20T23:52:50.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 218559 10.02 MB/sec execute 394 sec latency 98.176 ms 2024-09-20T23:52:50.469 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:50 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 96/1575 objects degraded (6.095%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:50 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 96/1575 objects degraded (6.095%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:50.976 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:50 smithi137 podman[78894]: 2024-09-20 23:52:50.869797812 +0000 UTC m=+1.593655174 container remove bc8f45f570f8916e4a9ac738e56fa4a5c1f7936b78c8fef4fbb375321a81f0ff (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T23:52:50.976 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:50 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.3.service: Deactivated successfully. 2024-09-20T23:52:50.976 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:50 smithi137 systemd[1]: Stopped Ceph osd.3 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:52:50.976 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:50 smithi137 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.3.service: Consumed 4min 18.400s CPU time. 2024-09-20T23:52:51.227 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 systemd[1]: Starting Ceph osd.3 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:52:51.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 218962 10.02 MB/sec execute 395 sec latency 54.048 ms 2024-09-20T23:52:51.481 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:51 smithi137 ceph-mon[65321]: pgmap v165: 97 pgs: 8 active+undersized, 9 peering, 3 stale+active+clean, 15 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 96/1575 objects degraded (6.095%) 2024-09-20T23:52:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:51 smithi144 ceph-mon[51829]: pgmap v165: 97 pgs: 8 active+undersized, 9 peering, 3 stale+active+clean, 15 active+undersized+degraded, 62 active+clean; 337 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 4.6 MiB/s wr, 440 op/s; 96/1575 objects degraded (6.095%) 2024-09-20T23:52:51.731 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 podman[78990]: 2024-09-20 23:52:51.38382981 +0000 UTC m=+0.020055620 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:51.732 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 podman[78990]: 2024-09-20 23:52:51.507023399 +0000 UTC m=+0.143249202 container create d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T23:52:51.991 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 podman[78990]: 2024-09-20 23:52:51.871863307 +0000 UTC m=+0.508089124 container init d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T23:52:51.991 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 podman[78990]: 2024-09-20 23:52:51.884592188 +0000 UTC m=+0.520817996 container start d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True) 2024-09-20T23:52:51.991 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:51 smithi137 podman[78990]: 2024-09-20 23:52:51.980007338 +0000 UTC m=+0.616233141 container attach d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:52:52.241 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:52.242 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:52.242 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:52.242 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:52:52.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 219248 10.02 MB/sec execute 396 sec latency 70.185 ms 2024-09-20T23:52:52.494 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-mon[65321]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-20T23:52:52.745 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T23:52:52.745 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T23:52:52.745 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T23:52:52.745 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T23:52:52.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:52 smithi144 ceph-mon[51829]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-09-20T23:52:52.998 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T23:52:52.999 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-09-20T23:52:52.999 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate[79004]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T23:52:52.999 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:52 smithi137 bash[78990]: --> ceph-volume raw activate successful for osd ID: 3 2024-09-20T23:52:53.255 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:53 smithi137 podman[78990]: 2024-09-20 23:52:53.027694921 +0000 UTC m=+1.663920733 container died d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731) 2024-09-20T23:52:53.274 INFO:teuthology.orchestra.run.smithi137.stdout: 5 219571 10.02 MB/sec execute 397 sec latency 75.465 ms 2024-09-20T23:52:53.506 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:53 smithi137 ceph-mon[65321]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.9 MiB/s wr, 419 op/s; 193/1575 objects degraded (12.254%) 2024-09-20T23:52:53.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:53 smithi144 ceph-mon[51829]: pgmap v166: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 336 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.9 MiB/s wr, 419 op/s; 193/1575 objects degraded (12.254%) 2024-09-20T23:52:54.032 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:53 smithi137 podman[79141]: 2024-09-20 23:52:53.836355459 +0000 UTC m=+0.799534302 container remove d9bc0d31d9a5a94e173e5799f95c08b3e719a28f87d8eac6cfa79d2119037b55 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-activate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True) 2024-09-20T23:52:54.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 219827 10.01 MB/sec execute 398 sec latency 54.992 ms 2024-09-20T23:52:54.288 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 podman[79199]: 2024-09-20 23:52:53.974329188 +0000 UTC m=+0.022408631 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:52:54.288 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 podman[79199]: 2024-09-20 23:52:54.104329753 +0000 UTC m=+0.152409185 container create c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True) 2024-09-20T23:52:54.816 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 podman[79199]: 2024-09-20 23:52:54.566821256 +0000 UTC m=+0.614900684 container init c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731) 2024-09-20T23:52:54.816 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 podman[79199]: 2024-09-20 23:52:54.57379406 +0000 UTC m=+0.621873496 container start c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T23:52:54.816 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 ceph-osd[79217]: -- 172.21.15.137:0/2373576491 <== mon.0 v2:172.21.15.137:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5635908d01e0 con 0x56358fabdc00 2024-09-20T23:52:55.066 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 bash[79199]: c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 2024-09-20T23:52:55.067 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:54 smithi137 systemd[1]: Started Ceph osd.3 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:52:55.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 220074 10.00 MB/sec execute 399 sec latency 83.825 ms 2024-09-20T23:52:55.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:55 smithi137 ceph-mon[65321]: pgmap v167: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 508 op/s; 206/1659 objects degraded (12.417%) 2024-09-20T23:52:55.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:55 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:55.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:55 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:55.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:55 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:55 smithi144 ceph-mon[51829]: pgmap v167: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.7 MiB/s wr, 508 op/s; 206/1659 objects degraded (12.417%) 2024-09-20T23:52:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:55 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:55 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:52:55.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:55 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:52:56.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 220320 10.00 MB/sec execute 400 sec latency 45.149 ms 2024-09-20T23:52:56.620 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:56 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 206/1659 objects degraded (12.417%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:56.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:56 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 206/1659 objects degraded (12.417%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:52:57.142 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:52:56 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:52:56.895+0000 7fb70a223640 -1 Falling back to public interface 2024-09-20T23:52:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 220533 9.99 MB/sec execute 401 sec latency 52.254 ms 2024-09-20T23:52:57.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:57 smithi137 ceph-mon[65321]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 425 op/s; 208/1656 objects degraded (12.560%) 2024-09-20T23:52:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:57 smithi144 ceph-mon[51829]: pgmap v168: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.1 MiB/s wr, 425 op/s; 208/1656 objects degraded (12.560%) 2024-09-20T23:52:58.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 220913 9.99 MB/sec execute 402 sec latency 40.966 ms 2024-09-20T23:52:59.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 221294 9.97 MB/sec execute 403 sec latency 51.972 ms 2024-09-20T23:52:59.675 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:52:59 smithi137 ceph-mon[65321]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 383 op/s; 205/1641 objects degraded (12.492%) 2024-09-20T23:52:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:52:59 smithi144 ceph-mon[51829]: pgmap v169: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.1 KiB/s rd, 3.7 MiB/s wr, 383 op/s; 205/1641 objects degraded (12.492%) 2024-09-20T23:53:00.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 221553 9.96 MB/sec execute 404 sec latency 75.546 ms 2024-09-20T23:53:01.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 221949 9.97 MB/sec execute 405 sec latency 79.694 ms 2024-09-20T23:53:01.497 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:01 smithi137 ceph-mon[65321]: pgmap v170: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 509 op/s; 201/1662 objects degraded (12.094%) 2024-09-20T23:53:01.498 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:01 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 201/1662 objects degraded (12.094%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:01.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:01 smithi144 ceph-mon[51829]: pgmap v170: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 342 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 7.2 KiB/s rd, 4.7 MiB/s wr, 509 op/s; 201/1662 objects degraded (12.094%) 2024-09-20T23:53:01.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:01 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 201/1662 objects degraded (12.094%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:02.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 222465 9.98 MB/sec execute 406 sec latency 36.267 ms 2024-09-20T23:53:03.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 222752 9.98 MB/sec execute 407 sec latency 54.292 ms 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: pgmap v171: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.1 MiB/s wr, 393 op/s; 207/1686 objects degraded (12.278%) 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.802 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: pgmap v171: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.5 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.1 MiB/s wr, 393 op/s; 207/1686 objects degraded (12.278%) 2024-09-20T23:53:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:04.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 223018 9.97 MB/sec execute 408 sec latency 50.419 ms 2024-09-20T23:53:04.578 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:53:04 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:53:04.512+0000 7fb70a223640 -1 osd.3 0 read_superblock omap replica is missing. 2024-09-20T23:53:05.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 223262 9.97 MB/sec execute 409 sec latency 66.820 ms 2024-09-20T23:53:05.878 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:05 smithi137 ceph-mon[65321]: pgmap v172: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 211/1668 objects degraded (12.650%) 2024-09-20T23:53:05.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:05 smithi144 ceph-mon[51829]: pgmap v172: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 341 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 8.3 KiB/s rd, 5.4 MiB/s wr, 556 op/s; 211/1668 objects degraded (12.650%) 2024-09-20T23:53:06.135 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:53:05 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:53:05.886+0000 7fb70a223640 -1 osd.3 112 log_to_monitors true 2024-09-20T23:53:06.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 223506 9.95 MB/sec execute 410 sec latency 51.209 ms 2024-09-20T23:53:06.961 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:06 smithi137 ceph-mon[65321]: from='osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:53:06.961 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:06 smithi137 ceph-mon[65321]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:53:06.961 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:06 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 211/1668 objects degraded (12.650%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:06 smithi144 ceph-mon[51829]: from='osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:53:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:06 smithi144 ceph-mon[51829]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-20T23:53:07.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:06 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 211/1668 objects degraded (12.650%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:07.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 223799 9.96 MB/sec execute 411 sec latency 79.581 ms 2024-09-20T23:53:07.462 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:53:07.209+0000 7fb6fdfc6640 -1 osd.3 112 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:53:07.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: pgmap v173: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.1 MiB/s wr, 407 op/s; 209/1659 objects degraded (12.598%) 2024-09-20T23:53:07.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T23:53:07.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: osdmap e115: 8 total, 7 up, 8 in 2024-09-20T23:53:07.714 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: from='osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:53:07.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:53:07.715 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:07 smithi137 ceph-mon[65321]: from='osd.3 ' entity='osd.3' 2024-09-20T23:53:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: pgmap v173: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.6 GiB used, 712 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.1 MiB/s wr, 407 op/s; 209/1659 objects degraded (12.598%) 2024-09-20T23:53:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-20T23:53:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: osdmap e115: 8 total, 7 up, 8 in 2024-09-20T23:53:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: from='osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:53:08.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-20T23:53:08.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:07 smithi144 ceph-mon[51829]: from='osd.3 ' entity='osd.3' 2024-09-20T23:53:08.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 224268 9.97 MB/sec execute 412 sec latency 71.375 ms 2024-09-20T23:53:08.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:08 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:53:08.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:08 smithi137 ceph-mon[65321]: osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189] boot 2024-09-20T23:53:08.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:08 smithi137 ceph-mon[65321]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T23:53:08.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:53:09.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:08 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:53:09.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:08 smithi144 ceph-mon[51829]: osd.3 [v2:172.21.15.137:6826/3608971189,v1:172.21.15.137:6827/3608971189] boot 2024-09-20T23:53:09.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:08 smithi144 ceph-mon[51829]: osdmap e116: 8 total, 8 up, 8 in 2024-09-20T23:53:09.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-20T23:53:09.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 224420 9.95 MB/sec execute 413 sec latency 1071.408 ms 2024-09-20T23:53:10.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 352 op/s; 206/1647 objects degraded (12.508%) 2024-09-20T23:53:10.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:10.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: osdmap e117: 8 total, 8 up, 8 in 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: pgmap v176: 97 pgs: 14 active+undersized, 27 active+undersized+degraded, 56 active+clean; 339 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.7 MiB/s wr, 352 op/s; 206/1647 objects degraded (12.508%) 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: osdmap e117: 8 total, 8 up, 8 in 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:10.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:10.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 224440 9.92 MB/sec execute 414 sec latency 2040.964 ms 2024-09-20T23:53:10.771 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:10 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:10.772 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:10 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:53:10.772 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:10 smithi137 ceph-mon[65321]: osdmap e118: 8 total, 8 up, 8 in 2024-09-20T23:53:11.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:10 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:11.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:10 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:53:11.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:10 smithi144 ceph-mon[51829]: osdmap e118: 8 total, 8 up, 8 in 2024-09-20T23:53:11.275 INFO:teuthology.orchestra.run.smithi137.stdout: 5 224772 9.92 MB/sec execute 415 sec latency 41.884 ms 2024-09-20T23:53:11.788 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:11 smithi137 ceph-mon[65321]: pgmap v179: 97 pgs: 1 remapped+peering, 3 activating, 10 peering, 8 active+undersized, 19 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 288 op/s; 136/1698 objects degraded (8.009%) 2024-09-20T23:53:11.788 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:11 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 206/1647 objects degraded (12.508%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:11.788 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:11 smithi137 ceph-mon[65321]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T23:53:12.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:11 smithi144 ceph-mon[51829]: pgmap v179: 97 pgs: 1 remapped+peering, 3 activating, 10 peering, 8 active+undersized, 19 active+undersized+degraded, 56 active+clean; 344 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 4.5 KiB/s rd, 3.6 MiB/s wr, 288 op/s; 136/1698 objects degraded (8.009%) 2024-09-20T23:53:12.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:11 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 206/1647 objects degraded (12.508%), 27 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:12.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:11 smithi144 ceph-mon[51829]: osdmap e119: 8 total, 8 up, 8 in 2024-09-20T23:53:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 224981 9.90 MB/sec execute 416 sec latency 953.752 ms 2024-09-20T23:53:13.213 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:53:13.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 225334 9.91 MB/sec execute 417 sec latency 888.169 ms 2024-09-20T23:53:13.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:13 smithi137 ceph-mon[65321]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T23:53:13.405 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:13 smithi137 ceph-mon[65321]: pgmap v182: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 activating, 15 peering, 1 active+undersized+degraded, 76 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.6 MiB/s wr, 585 op/s; 66/1683 objects degraded (3.922%); 182 KiB/s, 30 objects/s recovering 2024-09-20T23:53:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:13 smithi144 ceph-mon[51829]: osdmap e120: 8 total, 8 up, 8 in 2024-09-20T23:53:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:13 smithi144 ceph-mon[51829]: pgmap v182: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 3 activating, 15 peering, 1 active+undersized+degraded, 76 active+clean; 343 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 5.6 MiB/s wr, 585 op/s; 66/1683 objects degraded (3.922%); 182 KiB/s, 30 objects/s recovering 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (6m) 11s ago 15m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (3m) 11s ago 15m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (3m) 3m ago 14m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (6m) 11s ago 15m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (9m) 11s ago 9m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (9m) 3m ago 9m 14.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (5m) 11s ago 16m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (4m) 3m ago 14m 439M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (4m) 11s ago 16m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:53:14.201 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (4m) 3m ago 14m 36.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (9m) 11s ago 9m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (9m) 3m ago 9m 83.6M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 11s ago 15m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (5m) 3m ago 14m 16.6M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (2m) 11s ago 13m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (114s) 11s ago 13m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (65s) 11s ago 13m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (20s) 11s ago 12m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (12m) 3m ago 12m 533M 2704M 17.2.0 e1d6a67b021e 98460f602a46 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (11m) 3m ago 11m 337M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (11m) 3m ago 11m 532M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (10m) 3m ago 10m 444M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:53:14.202 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (4m) 11s ago 14m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:53:14.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 225737 9.90 MB/sec execute 418 sec latency 1022.564 ms 2024-09-20T23:53:14.451 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:14 smithi137 ceph-mon[65321]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:14.451 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:14 smithi137 ceph-mon[65321]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T23:53:14.451 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:14 smithi137 ceph-mon[65321]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:14 smithi144 ceph-mon[51829]: from='client.34314 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:14 smithi144 ceph-mon[51829]: osdmap e121: 8 total, 8 up, 8 in 2024-09-20T23:53:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:14 smithi144 ceph-mon[51829]: from='client.34318 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:14.916 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:53:14.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:14.918 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:53:14.918 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 6, 2024-09-20T23:53:14.918 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:53:14.918 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:53:14.918 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:53:15.237 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:15 smithi137 ceph-mon[65321]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:15.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:15 smithi137 ceph-mon[65321]: osdmap e122: 8 total, 8 up, 8 in 2024-09-20T23:53:15.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:15 smithi137 ceph-mon[65321]: pgmap v185: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 1 active+undersized+degraded, 89 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.8 MiB/s wr, 668 op/s; 65/1578 objects degraded (4.119%); 210 KiB/s, 46 objects/s recovering 2024-09-20T23:53:15.238 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:15 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/7652712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:15.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 226196 9.90 MB/sec execute 419 sec latency 43.470 ms 2024-09-20T23:53:15.410 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:53:15.410 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "10/23 daemons upgraded", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:53:15.411 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:53:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:15 smithi144 ceph-mon[51829]: from='client.34322 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:15 smithi144 ceph-mon[51829]: osdmap e122: 8 total, 8 up, 8 in 2024-09-20T23:53:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:15 smithi144 ceph-mon[51829]: pgmap v185: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 5 peering, 1 active+undersized+degraded, 89 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 11 KiB/s rd, 4.8 MiB/s wr, 668 op/s; 65/1578 objects degraded (4.119%); 210 KiB/s, 46 objects/s recovering 2024-09-20T23:53:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:15 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/7652712' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:16.057 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN Degraded data redundancy: 65/1578 objects degraded (4.119%), 3 pgs degraded 2024-09-20T23:53:16.057 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 65/1578 objects degraded (4.119%), 3 pgs degraded 2024-09-20T23:53:16.057 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.2 is active+undersized+degraded, acting [5,6] 2024-09-20T23:53:16.057 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.12 is active+recovery_wait+undersized+degraded+remapped, acting [0,7] 2024-09-20T23:53:16.057 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.17 is active+recovery_wait+undersized+degraded+remapped, acting [0,5] 2024-09-20T23:53:16.249 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:16 smithi137 ceph-mon[65321]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:16 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2380791446' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:53:16.250 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:16 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 65/1578 objects degraded (4.119%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:16.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 226531 9.91 MB/sec execute 420 sec latency 33.416 ms 2024-09-20T23:53:16.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:16 smithi144 ceph-mon[51829]: from='client.34330 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:16.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:16 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2380791446' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:53:16.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:16 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 65/1578 objects degraded (4.119%), 3 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:17.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 226949 9.92 MB/sec execute 421 sec latency 35.379 ms 2024-09-20T23:53:17.529 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:17 smithi137 ceph-mon[65321]: pgmap v186: 97 pgs: 5 peering, 1 active+undersized+degraded, 91 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 533 op/s; 14/1554 objects degraded (0.901%); 343 KiB/s, 42 objects/s recovering 2024-09-20T23:53:17.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:17 smithi144 ceph-mon[51829]: pgmap v186: 97 pgs: 5 peering, 1 active+undersized+degraded, 91 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.7 MiB/s wr, 533 op/s; 14/1554 objects degraded (0.901%); 343 KiB/s, 42 objects/s recovering 2024-09-20T23:53:18.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 227288 9.93 MB/sec execute 422 sec latency 71.269 ms 2024-09-20T23:53:19.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 227469 9.92 MB/sec execute 423 sec latency 66.211 ms 2024-09-20T23:53:19.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:19.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:19.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:19 smithi137 ceph-mon[65321]: pgmap v187: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 417 op/s; 279 KiB/s, 30 objects/s recovering 2024-09-20T23:53:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:19 smithi144 ceph-mon[51829]: pgmap v187: 97 pgs: 97 active+clean; 342 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.0 MiB/s wr, 417 op/s; 279 KiB/s, 30 objects/s recovering 2024-09-20T23:53:20.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 227646 9.90 MB/sec execute 424 sec latency 74.938 ms 2024-09-20T23:53:20.323 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:20 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1554 objects degraded (0.901%), 1 pg degraded) 2024-09-20T23:53:20.323 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:20 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:53:20.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:20 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 14/1554 objects degraded (0.901%), 1 pg degraded) 2024-09-20T23:53:20.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:20 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:53:21.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 227842 9.89 MB/sec execute 425 sec latency 76.022 ms 2024-09-20T23:53:21.363 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:21 smithi137 ceph-mon[65321]: pgmap v188: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 436 op/s; 210 KiB/s, 22 objects/s recovering 2024-09-20T23:53:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:21 smithi144 ceph-mon[51829]: pgmap v188: 97 pgs: 97 active+clean; 340 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.8 MiB/s wr, 436 op/s; 210 KiB/s, 22 objects/s recovering 2024-09-20T23:53:22.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 228106 9.87 MB/sec execute 426 sec latency 79.470 ms 2024-09-20T23:53:23.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 228403 9.86 MB/sec execute 427 sec latency 61.853 ms 2024-09-20T23:53:23.411 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:23 smithi137 ceph-mon[65321]: pgmap v189: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 471 op/s; 187 KiB/s, 20 objects/s recovering 2024-09-20T23:53:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:23 smithi144 ceph-mon[51829]: pgmap v189: 97 pgs: 97 active+clean; 343 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 5.3 MiB/s wr, 471 op/s; 187 KiB/s, 20 objects/s recovering 2024-09-20T23:53:24.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 228704 9.86 MB/sec execute 428 sec latency 55.426 ms 2024-09-20T23:53:24.453 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:24.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:25.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 229046 9.86 MB/sec execute 429 sec latency 75.564 ms 2024-09-20T23:53:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:25 smithi144 ceph-mon[51829]: pgmap v190: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 379 op/s; 157 KiB/s, 11 objects/s recovering 2024-09-20T23:53:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:25.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:25 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:25.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:25 smithi144 ceph-mon[51829]: Upgrade: osd.4 is safe to restart 2024-09-20T23:53:25.470 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:25 smithi137 ceph-mon[65321]: pgmap v190: 97 pgs: 97 active+clean; 338 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.6 MiB/s wr, 379 op/s; 157 KiB/s, 11 objects/s recovering 2024-09-20T23:53:25.470 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:25.470 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:25 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["4"], "max": 16}]: dispatch 2024-09-20T23:53:25.470 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:25 smithi137 ceph-mon[65321]: Upgrade: osd.4 is safe to restart 2024-09-20T23:53:26.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 229450 9.85 MB/sec execute 430 sec latency 45.145 ms 2024-09-20T23:53:26.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:26 smithi137 ceph-mon[65321]: Upgrade: Updating osd.4 2024-09-20T23:53:26.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:26 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:26.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:26 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:53:26.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:26 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:26.978 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:26 smithi137 ceph-mon[65321]: Deploying daemon osd.4 on smithi144 2024-09-20T23:53:27.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 229816 9.85 MB/sec execute 431 sec latency 61.590 ms 2024-09-20T23:53:27.429 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:26 smithi144 ceph-mon[51829]: Upgrade: Updating osd.4 2024-09-20T23:53:27.429 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:26 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:27.429 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:26 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-20T23:53:27.429 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:26 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:27.429 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:26 smithi144 ceph-mon[51829]: Deploying daemon osd.4 on smithi144 2024-09-20T23:53:28.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:27 smithi137 ceph-mon[65321]: pgmap v191: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 325 op/s; 131 KiB/s, 9 objects/s recovering 2024-09-20T23:53:28.025 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:27 smithi144 ceph-mon[51829]: pgmap v191: 97 pgs: 97 active+clean; 337 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.0 MiB/s wr, 325 op/s; 131 KiB/s, 9 objects/s recovering 2024-09-20T23:53:28.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 230135 9.86 MB/sec execute 432 sec latency 50.016 ms 2024-09-20T23:53:29.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 230503 9.87 MB/sec execute 433 sec latency 42.297 ms 2024-09-20T23:53:29.328 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:29 smithi144 systemd[1]: Stopping Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:53:29.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:29 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[30264]: 2024-09-20T23:53:29.342+0000 7fa1bd1f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:53:29.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:29 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[30264]: 2024-09-20T23:53:29.343+0000 7fa1bd1f7700 -1 osd.4 122 *** Got signal Terminated *** 2024-09-20T23:53:29.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:29 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[30264]: 2024-09-20T23:53:29.343+0000 7fa1bd1f7700 -1 osd.4 122 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:53:30.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:29 smithi137 ceph-mon[65321]: pgmap v192: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 439 op/s; 43 KiB/s, 5 objects/s recovering 2024-09-20T23:53:30.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:29 smithi137 ceph-mon[65321]: osd.4 marked itself down and dead 2024-09-20T23:53:30.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 230748 9.87 MB/sec execute 434 sec latency 305.015 ms 2024-09-20T23:53:30.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:29 smithi144 ceph-mon[51829]: pgmap v192: 97 pgs: 97 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 5.9 KiB/s rd, 4.4 MiB/s wr, 439 op/s; 43 KiB/s, 5 objects/s recovering 2024-09-20T23:53:30.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:29 smithi144 ceph-mon[51829]: osd.4 marked itself down and dead 2024-09-20T23:53:30.895 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:30 smithi144 podman[56240]: 2024-09-20 23:53:30.606183272 +0000 UTC m=+1.390546914 container died 98460f602a4657106c602f00196e8aeeedcfff96828182621c9aadce162915e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, ceph=True, GIT_BRANCH=HEAD, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.buildah.version=1.19.8, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-09-20T23:53:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:30 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:53:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:30 smithi144 ceph-mon[51829]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T23:53:31.194 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:31 smithi144 podman[56240]: 2024-09-20 23:53:31.085402851 +0000 UTC m=+1.869766482 container cleanup 98460f602a4657106c602f00196e8aeeedcfff96828182621c9aadce162915e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, GIT_BRANCH=HEAD, RELEASE=HEAD, vcs-type=git, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.tags=base centos centos-stream, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, architecture=x86_64, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8) 2024-09-20T23:53:31.194 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:31 smithi144 bash[56240]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4 2024-09-20T23:53:31.221 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:30 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:53:31.222 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:30 smithi137 ceph-mon[65321]: osdmap e123: 8 total, 7 up, 8 in 2024-09-20T23:53:31.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 230828 9.85 MB/sec execute 435 sec latency 1049.564 ms 2024-09-20T23:53:31.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:31 smithi144 podman[56255]: 2024-09-20 23:53:31.370845831 +0000 UTC m=+0.768364259 container remove 98460f602a4657106c602f00196e8aeeedcfff96828182621c9aadce162915e9 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, name=centos-stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, io.openshift.expose-services=, RELEASE=HEAD, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=-17.2.0, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., vcs-type=git, com.redhat.component=centos-stream-container) 2024-09-20T23:53:31.986 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:31 smithi144 podman[56345]: 2024-09-20 23:53:31.637130928 +0000 UTC m=+0.017509104 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:53:31.986 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:31 smithi144 podman[56345]: 2024-09-20 23:53:31.763493158 +0000 UTC m=+0.143871322 container create 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-09-20T23:53:31.990 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:31 smithi137 ceph-mon[65321]: pgmap v194: 97 pgs: 15 stale+active+clean, 82 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.3 MiB/s wr, 487 op/s 2024-09-20T23:53:31.990 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:31 smithi137 ceph-mon[65321]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T23:53:32.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 231103 9.85 MB/sec execute 436 sec latency 51.506 ms 2024-09-20T23:53:32.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:31 smithi144 ceph-mon[51829]: pgmap v194: 97 pgs: 15 stale+active+clean, 82 active+clean; 336 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.3 MiB/s wr, 487 op/s 2024-09-20T23:53:32.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:31 smithi144 ceph-mon[51829]: osdmap e124: 8 total, 7 up, 8 in 2024-09-20T23:53:32.444 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:32 smithi144 podman[56345]: 2024-09-20 23:53:32.024760734 +0000 UTC m=+0.405138908 container init 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-20T23:53:32.444 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:32 smithi144 podman[56345]: 2024-09-20 23:53:32.028598328 +0000 UTC m=+0.408976497 container start 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-20T23:53:32.444 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:32 smithi144 podman[56345]: 2024-09-20 23:53:32.126022853 +0000 UTC m=+0.506401025 container attach 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T23:53:32.444 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:32 smithi144 podman[56345]: 2024-09-20 23:53:32.232467532 +0000 UTC m=+0.612845702 container died 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:53:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:33 smithi144 ceph-mon[51829]: pgmap v196: 97 pgs: 7 active+undersized, 8 peering, 8 stale+active+clean, 5 active+undersized+degraded, 69 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.6 MiB/s wr, 486 op/s; 54/1680 objects degraded (3.214%) 2024-09-20T23:53:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:33 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 54/1680 objects degraded (3.214%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:33.274 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:33 smithi137 ceph-mon[65321]: pgmap v196: 97 pgs: 7 active+undersized, 8 peering, 8 stale+active+clean, 5 active+undersized+degraded, 69 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.4 KiB/s rd, 4.6 MiB/s wr, 486 op/s; 54/1680 objects degraded (3.214%) 2024-09-20T23:53:33.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:33 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 54/1680 objects degraded (3.214%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:33.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 231415 9.84 MB/sec execute 437 sec latency 42.868 ms 2024-09-20T23:53:33.906 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 podman[56368]: 2024-09-20 23:53:33.448007164 +0000 UTC m=+1.208897632 container remove 0d0fc82620e659b5e05e1bee831624213ee0ce5b4ce290ad1d1207f9a04cb243 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-09-20T23:53:33.906 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-20T23:53:33.906 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 systemd[1]: Stopped Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:53:33.906 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service: Consumed 4min 11.756s CPU time. 2024-09-20T23:53:33.907 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 systemd[1]: Starting Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:53:34.193 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 podman[56468]: 2024-09-20 23:53:33.807948115 +0000 UTC m=+0.018834168 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:53:34.193 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:33 smithi144 podman[56468]: 2024-09-20 23:53:33.924036711 +0000 UTC m=+0.134922752 container create 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:53:34.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 231718 9.82 MB/sec execute 438 sec latency 78.404 ms 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:34.286 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:34.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:34.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:34.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 podman[56468]: 2024-09-20 23:53:34.193923971 +0000 UTC m=+0.404810013 container init 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-20T23:53:34.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 podman[56468]: 2024-09-20 23:53:34.197652988 +0000 UTC m=+0.408539034 container start 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T23:53:34.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 podman[56468]: 2024-09-20 23:53:34.294226787 +0000 UTC m=+0.505112846 container attach 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True) 2024-09-20T23:53:34.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:53:34.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 bash[56468]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:53:34.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:53:34.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:34 smithi144 bash[56468]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:53:35.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 231977 9.82 MB/sec execute 439 sec latency 86.325 ms 2024-09-20T23:53:35.317 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:35 smithi137 ceph-mon[65321]: pgmap v197: 97 pgs: 12 active+undersized, 8 peering, 22 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 516 op/s; 209/1719 objects degraded (12.158%) 2024-09-20T23:53:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:35 smithi144 ceph-mon[51829]: pgmap v197: 97 pgs: 12 active+undersized, 8 peering, 22 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 5.0 MiB/s wr, 516 op/s; 209/1719 objects degraded (12.158%) 2024-09-20T23:53:36.028 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:35 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T23:53:36.029 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:35 smithi144 bash[56468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T23:53:36.029 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:35 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T23:53:36.029 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:35 smithi144 bash[56468]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-4 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-20T23:53:36.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 232282 9.81 MB/sec execute 440 sec latency 57.179 ms 2024-09-20T23:53:36.365 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 bash[56468]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 bash[56468]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 bash[56468]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-4/block 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 bash[56468]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate[56484]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 bash[56468]: --> ceph-volume raw activate successful for osd ID: 4 2024-09-20T23:53:36.366 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 podman[56468]: 2024-09-20 23:53:36.065923072 +0000 UTC m=+2.276809123 container died 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T23:53:36.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 podman[56621]: 2024-09-20 23:53:36.643771901 +0000 UTC m=+0.571723379 container remove 8b71e3d34725a0e8dba7d6a18af9240b50865b5d35e330585f6436ca04ac2ff6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731) 2024-09-20T23:53:36.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 podman[56679]: 2024-09-20 23:53:36.760500443 +0000 UTC m=+0.018279684 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:53:36.944 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:36 smithi144 podman[56679]: 2024-09-20 23:53:36.886550603 +0000 UTC m=+0.144329826 container create 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-20T23:53:37.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 232639 9.81 MB/sec execute 441 sec latency 65.338 ms 2024-09-20T23:53:37.281 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:37 smithi144 podman[56679]: 2024-09-20 23:53:37.132009063 +0000 UTC m=+0.389788301 container init 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9e2ac70) 2024-09-20T23:53:37.281 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:37 smithi144 podman[56679]: 2024-09-20 23:53:37.135653227 +0000 UTC m=+0.393432454 container start 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T23:53:37.281 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:37 smithi144 bash[56679]: 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 2024-09-20T23:53:37.281 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:37 smithi144 systemd[1]: Started Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:53:37.369 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:37 smithi137 ceph-mon[65321]: pgmap v198: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 344 op/s; 269/1707 objects degraded (15.759%) 2024-09-20T23:53:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:37 smithi144 ceph-mon[51829]: pgmap v198: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 343 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 344 op/s; 269/1707 objects degraded (15.759%) 2024-09-20T23:53:38.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 232994 9.80 MB/sec execute 442 sec latency 58.820 ms 2024-09-20T23:53:38.370 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:38.370 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:38.370 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:38.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:39.276 INFO:teuthology.orchestra.run.smithi137.stdout: 5 233321 9.80 MB/sec execute 443 sec latency 83.387 ms 2024-09-20T23:53:39.306 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:39 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:53:39.004+0000 7fc6efc3c640 -1 Falling back to public interface 2024-09-20T23:53:39.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:39 smithi137 ceph-mon[65321]: pgmap v199: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.7 MiB/s wr, 374 op/s; 263/1653 objects degraded (15.910%) 2024-09-20T23:53:39.405 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:39 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 263/1653 objects degraded (15.910%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:39.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:39 smithi144 ceph-mon[51829]: pgmap v199: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 3.8 KiB/s rd, 3.7 MiB/s wr, 374 op/s; 263/1653 objects degraded (15.910%) 2024-09-20T23:53:39.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:39 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 263/1653 objects degraded (15.910%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:40.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 233562 9.80 MB/sec execute 444 sec latency 65.316 ms 2024-09-20T23:53:41.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 233898 9.81 MB/sec execute 445 sec latency 51.375 ms 2024-09-20T23:53:41.459 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:41 smithi137 ceph-mon[65321]: pgmap v200: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.9 MiB/s wr, 410 op/s; 252/1569 objects degraded (16.061%) 2024-09-20T23:53:41.459 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:41.459 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:41 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:41 smithi144 ceph-mon[51829]: pgmap v200: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 331 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.9 MiB/s wr, 410 op/s; 252/1569 objects degraded (16.061%) 2024-09-20T23:53:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:41 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:42.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 234196 9.81 MB/sec execute 446 sec latency 56.840 ms 2024-09-20T23:53:43.002 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:42 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:43.003 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:42 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:43.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:42 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:43.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:42 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 234464 9.80 MB/sec execute 447 sec latency 86.243 ms 2024-09-20T23:53:44.061 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:43 smithi137 ceph-mon[65321]: pgmap v201: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 407 op/s; 258/1605 objects degraded (16.075%) 2024-09-20T23:53:44.102 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:43 smithi144 ceph-mon[51829]: pgmap v201: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 335 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.3 MiB/s wr, 407 op/s; 258/1605 objects degraded (16.075%) 2024-09-20T23:53:44.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 234718 9.80 MB/sec execute 448 sec latency 62.789 ms 2024-09-20T23:53:45.077 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:44 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 258/1614 objects degraded (15.985%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:45.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 234985 9.79 MB/sec execute 449 sec latency 47.080 ms 2024-09-20T23:53:45.397 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:44 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 258/1614 objects degraded (15.985%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: pgmap v202: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.4 MiB/s wr, 472 op/s; 258/1614 objects degraded (15.985%) 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:46.081 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.082 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.082 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.082 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:46 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:53:46.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 235286 9.77 MB/sec execute 450 sec latency 75.745 ms 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: pgmap v202: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 338 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 8.0 KiB/s rd, 4.4 MiB/s wr, 472 op/s; 258/1614 objects degraded (15.985%) 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:53:46.366 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:46.367 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:53:46.537 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:53:46.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:46 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:53:46.363+0000 7fc6efc3c640 -1 osd.4 0 read_superblock omap replica is missing. 2024-09-20T23:53:47.164 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:47 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:53:47.164 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:47 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:53:47.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 235558 9.77 MB/sec execute 451 sec latency 62.823 ms 2024-09-20T23:53:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:47 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:53:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:47 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (7 PGs are or would become offline) 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (7m) 44s ago 16m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (3m) 44s ago 16m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (3m) 6s ago 15m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (7m) 44s ago 15m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (10m) 44s ago 10m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (10m) 6s ago 10m 14.8M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:53:47.598 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (6m) 44s ago 17m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (5m) 6s ago 15m 441M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (5m) 44s ago 17m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (4m) 6s ago 15m 41.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (9m) 44s ago 9m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (9m) 6s ago 9m 86.2M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (6m) 44s ago 15m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (6m) 6s ago 15m 16.8M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (3m) 44s ago 14m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (2m) 44s ago 13m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (99s) 44s ago 13m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (53s) 44s ago 13m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (10s) 6s ago 12m 12.0M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:53:47.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (12m) 6s ago 12m 548M 2704M 17.2.0 e1d6a67b021e 946b8c40c3ac 2024-09-20T23:53:47.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (11m) 6s ago 11m 469M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:53:47.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (11m) 6s ago 11m 725M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:53:47.600 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (5m) 44s ago 14m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:53:47.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:47 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:53:47.625+0000 7fc6efc3c640 -1 osd.4 122 log_to_monitors true 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:53:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 4 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 5, 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:53:48.132 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:53:48.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: pgmap v203: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.3 MiB/s wr, 456 op/s; 260/1635 objects degraded (15.902%) 2024-09-20T23:53:48.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: from='osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:53:48.261 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:48 smithi137 ceph-mon[65321]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:53:48.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 235755 9.76 MB/sec execute 452 sec latency 146.711 ms 2024-09-20T23:53:48.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: pgmap v203: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 339 MiB data, 3.4 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.3 MiB/s wr, 456 op/s; 260/1635 objects degraded (15.902%) 2024-09-20T23:53:48.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: from='client.34338 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: from='client.34342 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: from='client.34346 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:48.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: from='osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:53:48.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-mon[51829]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:53:48.562 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "11/23 daemons upgraded", 2024-09-20T23:53:48.563 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:53:48.563 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:53:48.563 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:53:48.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:53:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:53:48.500+0000 7fc6e39df640 -1 osd.4 122 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: osdmap e125: 8 total, 7 up, 8 in 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/1714805074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: pgmap v205: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 373 op/s; 265/1671 objects degraded (15.859%) 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='osd.4 ' entity='osd.4' 2024-09-20T23:53:49.270 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:49 smithi137 ceph-mon[65321]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:49.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 236160 9.76 MB/sec execute 453 sec latency 133.947 ms 2024-09-20T23:53:49.289 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 265/1671 objects degraded (15.859%), 29 pgs degraded 2024-09-20T23:53:49.289 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:53:49.289 INFO:teuthology.orchestra.run.smithi137.stdout: osd.4 (root=default,host=smithi144) is down 2024-09-20T23:53:49.289 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 265/1671 objects degraded (15.859%), 29 pgs degraded 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.5 is active+undersized+degraded, acting [7,0] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.6 is active+undersized+degraded, acting [1,6] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.b is active+undersized+degraded, acting [7,5] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.d is active+undersized+degraded, acting [1,3] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.f is active+undersized+degraded, acting [0,7] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.11 is active+undersized+degraded, acting [6,1] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.13 is active+undersized+degraded, acting [0,3] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.18 is active+undersized+degraded, acting [5,7] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1a is active+undersized+degraded, acting [6,7] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1c is active+undersized+degraded, acting [5,2] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1f is active+undersized+degraded, acting [0,7] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1 is active+undersized+degraded, acting [0,3] 2024-09-20T23:53:49.290 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.3 is active+undersized+degraded, acting [0,6] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.6 is active+undersized+degraded, acting [0,1] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.9 is active+undersized+degraded, acting [2,7] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.a is active+undersized+degraded, acting [6,1] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.b is active+undersized+degraded, acting [3,0] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.e is active+undersized+degraded, acting [7,1] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.f is active+undersized+degraded, acting [7,0] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.11 is active+undersized+degraded, acting [7,6] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.13 is active+undersized+degraded, acting [7,2] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.14 is active+undersized+degraded, acting [7,6] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.15 is active+undersized+degraded, acting [7,3] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.19 is active+undersized+degraded, acting [1,3] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1a is active+undersized+degraded, acting [1,2] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1b is active+undersized+degraded, acting [0,7] 2024-09-20T23:53:49.291 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1c is active+undersized+degraded, acting [5,1] 2024-09-20T23:53:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1d is active+undersized+degraded, acting [5,6] 2024-09-20T23:53:49.292 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.5 is active+undersized+degraded, acting [6,7] 2024-09-20T23:53:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-20T23:53:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: osdmap e125: 8 total, 7 up, 8 in 2024-09-20T23:53:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:53:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:53:49.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/1714805074' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:53:49.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:53:49.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: pgmap v205: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 340 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.4 MiB/s wr, 373 op/s; 265/1671 objects degraded (15.859%) 2024-09-20T23:53:49.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='osd.4 ' entity='osd.4' 2024-09-20T23:53:49.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:49 smithi144 ceph-mon[51829]: from='client.34356 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:53:50.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 236371 9.75 MB/sec execute 454 sec latency 767.927 ms 2024-09-20T23:53:50.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:50 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/634002365' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:53:50.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:50 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:53:50.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:50 smithi137 ceph-mon[65321]: osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162] boot 2024-09-20T23:53:50.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:50 smithi137 ceph-mon[65321]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T23:53:50.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:53:50.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:50 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/634002365' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:53:50.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:50 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:53:50.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:50 smithi144 ceph-mon[51829]: osd.4 [v2:172.21.15.144:6800/3487959162,v1:172.21.15.144:6801/3487959162] boot 2024-09-20T23:53:50.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:50 smithi144 ceph-mon[51829]: osdmap e126: 8 total, 8 up, 8 in 2024-09-20T23:53:50.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-20T23:53:51.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 236426 9.73 MB/sec execute 455 sec latency 1069.917 ms 2024-09-20T23:53:51.776 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:51 smithi137 ceph-mon[65321]: pgmap v207: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 493 op/s; 269/1671 objects degraded (16.098%) 2024-09-20T23:53:51.777 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:51 smithi137 ceph-mon[65321]: osdmap e127: 8 total, 8 up, 8 in 2024-09-20T23:53:51.777 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:51 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 269/1671 objects degraded (16.098%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:51 smithi144 ceph-mon[51829]: pgmap v207: 97 pgs: 13 active+undersized, 29 active+undersized+degraded, 55 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 8.7 KiB/s rd, 4.4 MiB/s wr, 493 op/s; 269/1671 objects degraded (16.098%) 2024-09-20T23:53:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:51 smithi144 ceph-mon[51829]: osdmap e127: 8 total, 8 up, 8 in 2024-09-20T23:53:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:51 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 269/1671 objects degraded (16.098%), 29 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:52.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 236696 9.72 MB/sec execute 456 sec latency 971.044 ms 2024-09-20T23:53:52.533 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:52 smithi137 ceph-mon[65321]: osdmap e128: 8 total, 8 up, 8 in 2024-09-20T23:53:52.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:52 smithi144 ceph-mon[51829]: osdmap e128: 8 total, 8 up, 8 in 2024-09-20T23:53:53.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 237087 9.73 MB/sec execute 457 sec latency 109.416 ms 2024-09-20T23:53:53.563 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:53 smithi137 ceph-mon[65321]: pgmap v210: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 6 peering, 9 active+undersized, 19 active+undersized+degraded, 60 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.8 MiB/s wr, 517 op/s; 205/1560 objects degraded (13.141%); 823 KiB/s, 44 keys/s, 5 objects/s recovering 2024-09-20T23:53:53.563 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:53 smithi137 ceph-mon[65321]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T23:53:53.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:53 smithi144 ceph-mon[51829]: pgmap v210: 97 pgs: 2 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 6 peering, 9 active+undersized, 19 active+undersized+degraded, 60 active+clean; 333 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 10 KiB/s rd, 3.8 MiB/s wr, 517 op/s; 205/1560 objects degraded (13.141%); 823 KiB/s, 44 keys/s, 5 objects/s recovering 2024-09-20T23:53:53.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:53 smithi144 ceph-mon[51829]: osdmap e129: 8 total, 8 up, 8 in 2024-09-20T23:53:54.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 237332 9.72 MB/sec execute 458 sec latency 703.636 ms 2024-09-20T23:53:54.791 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:54 smithi137 ceph-mon[65321]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T23:53:54.791 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:54 smithi137 ceph-mon[65321]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T23:53:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:54 smithi144 ceph-mon[51829]: osdmap e130: 8 total, 8 up, 8 in 2024-09-20T23:53:54.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:54 smithi144 ceph-mon[51829]: osdmap e131: 8 total, 8 up, 8 in 2024-09-20T23:53:55.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 237496 9.71 MB/sec execute 459 sec latency 975.405 ms 2024-09-20T23:53:55.790 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:55 smithi137 ceph-mon[65321]: pgmap v213: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 9 peering, 82 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.5 MiB/s wr, 392 op/s; 94/1611 objects degraded (5.835%); 1.8 MiB/s, 46 keys/s, 49 objects/s recovering 2024-09-20T23:53:55.829 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:55 smithi137 ceph-mon[65321]: osdmap e132: 8 total, 8 up, 8 in 2024-09-20T23:53:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:55 smithi144 ceph-mon[51829]: pgmap v213: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 9 peering, 82 active+clean; 339 MiB data, 3.0 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.5 MiB/s wr, 392 op/s; 94/1611 objects degraded (5.835%); 1.8 MiB/s, 46 keys/s, 49 objects/s recovering 2024-09-20T23:53:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:55 smithi144 ceph-mon[51829]: osdmap e132: 8 total, 8 up, 8 in 2024-09-20T23:53:56.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 237827 9.72 MB/sec execute 460 sec latency 953.921 ms 2024-09-20T23:53:56.549 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:56 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 94/1611 objects degraded (5.835%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:56.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:56 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 94/1611 objects degraded (5.835%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:53:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 238238 9.73 MB/sec execute 461 sec latency 35.172 ms 2024-09-20T23:53:57.559 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:57 smithi137 ceph-mon[65321]: pgmap v216: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 peering, 88 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 323 op/s; 94/1590 objects degraded (5.912%); 1.1 MiB/s, 57 objects/s recovering 2024-09-20T23:53:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:57 smithi144 ceph-mon[51829]: pgmap v216: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 4 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 3 peering, 88 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.8 MiB/s wr, 323 op/s; 94/1590 objects degraded (5.912%); 1.1 MiB/s, 57 objects/s recovering 2024-09-20T23:53:58.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 238535 9.72 MB/sec execute 462 sec latency 40.216 ms 2024-09-20T23:53:59.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 238840 9.71 MB/sec execute 463 sec latency 37.809 ms 2024-09-20T23:53:59.587 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:53:59 smithi137 ceph-mon[65321]: pgmap v217: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 92 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.0 MiB/s wr, 409 op/s; 47/1641 objects degraded (2.864%); 1.6 MiB/s, 48 objects/s recovering 2024-09-20T23:53:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:53:59 smithi144 ceph-mon[51829]: pgmap v217: 97 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovery_wait+undersized+degraded+remapped, 2 peering, 92 active+clean; 341 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 3.0 KiB/s rd, 4.0 MiB/s wr, 409 op/s; 47/1641 objects degraded (2.864%); 1.6 MiB/s, 48 objects/s recovering 2024-09-20T23:54:00.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 239107 9.70 MB/sec execute 464 sec latency 64.347 ms 2024-09-20T23:54:00.641 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:00 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/1641 objects degraded (2.864%), 3 pgs degraded) 2024-09-20T23:54:00.641 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:00 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:54:00.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:00 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 47/1641 objects degraded (2.864%), 3 pgs degraded) 2024-09-20T23:54:00.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:00 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:54:01.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 239448 9.70 MB/sec execute 465 sec latency 41.387 ms 2024-09-20T23:54:01.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:01 smithi137 ceph-mon[65321]: pgmap v218: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 390 op/s; 1.4 MiB/s, 40 objects/s recovering 2024-09-20T23:54:01.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:54:01.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:01 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:54:01.664 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:01 smithi137 ceph-mon[65321]: Upgrade: osd.5 is safe to restart 2024-09-20T23:54:01.667 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:01 smithi144 ceph-mon[51829]: pgmap v218: 97 pgs: 97 active+clean; 339 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.0 KiB/s rd, 4.4 MiB/s wr, 390 op/s; 1.4 MiB/s, 40 objects/s recovering 2024-09-20T23:54:01.667 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:01 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:54:01.667 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:01 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["5"], "max": 16}]: dispatch 2024-09-20T23:54:01.668 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:01 smithi144 ceph-mon[51829]: Upgrade: osd.5 is safe to restart 2024-09-20T23:54:02.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 239841 9.70 MB/sec execute 466 sec latency 48.460 ms 2024-09-20T23:54:03.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 240208 9.69 MB/sec execute 467 sec latency 56.187 ms 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: pgmap v219: 97 pgs: 97 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.5 MiB/s wr, 439 op/s; 661 KiB/s, 19 objects/s recovering 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: Upgrade: Updating osd.5 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: Deploying daemon osd.5 on smithi144 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:03.741 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: pgmap v219: 97 pgs: 97 active+clean; 345 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 4.0 KiB/s rd, 4.5 MiB/s wr, 439 op/s; 661 KiB/s, 19 objects/s recovering 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: Upgrade: Updating osd.5 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: Deploying daemon osd.5 on smithi144 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:03.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:04.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 240470 9.68 MB/sec execute 468 sec latency 79.118 ms 2024-09-20T23:54:05.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 240712 9.68 MB/sec execute 469 sec latency 57.546 ms 2024-09-20T23:54:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:05 smithi144 ceph-mon[51829]: pgmap v220: 97 pgs: 97 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.6 MiB/s wr, 416 op/s; 545 KiB/s, 15 objects/s recovering 2024-09-20T23:54:05.826 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:05 smithi137 ceph-mon[65321]: pgmap v220: 97 pgs: 97 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.6 MiB/s wr, 416 op/s; 545 KiB/s, 15 objects/s recovering 2024-09-20T23:54:06.174 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:05 smithi144 systemd[1]: Stopping Ceph osd.5 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:54:06.277 INFO:teuthology.orchestra.run.smithi137.stdout: 5 240953 9.68 MB/sec execute 470 sec latency 52.896 ms 2024-09-20T23:54:06.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[33207]: 2024-09-20T23:54:06.171+0000 7f80c050c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:54:06.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[33207]: 2024-09-20T23:54:06.171+0000 7f80c050c700 -1 osd.5 132 *** Got signal Terminated *** 2024-09-20T23:54:06.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[33207]: 2024-09-20T23:54:06.171+0000 7f80c050c700 -1 osd.5 132 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:54:06.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:06 smithi137 ceph-mon[65321]: osd.5 marked itself down and dead 2024-09-20T23:54:06.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:06 smithi144 ceph-mon[51829]: osd.5 marked itself down and dead 2024-09-20T23:54:07.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 241029 9.66 MB/sec execute 471 sec latency 702.436 ms 2024-09-20T23:54:07.539 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:07 smithi144 podman[58492]: 2024-09-20 23:54:07.231476927 +0000 UTC m=+1.183005544 container died 946b8c40c3ac593e3bbe230f4bdaa20d8fcfda9f5ed8070097ccf4f7849a0f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, ceph=True, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, maintainer=Guillaume Abrioux , release=754, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, GIT_CLEAN=True) 2024-09-20T23:54:07.539 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:07 smithi144 podman[58492]: 2024-09-20 23:54:07.538207567 +0000 UTC m=+1.489736186 container cleanup 946b8c40c3ac593e3bbe230f4bdaa20d8fcfda9f5ed8070097ccf4f7849a0f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, name=centos-stream, architecture=x86_64, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, io.buildah.version=1.19.8, vcs-type=git, vendor=Red Hat, Inc., distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-09-20T23:54:07.790 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:07 smithi144 bash[58492]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5 2024-09-20T23:54:07.791 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:07 smithi144 ceph-mon[51829]: pgmap v221: 97 pgs: 97 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.3 MiB/s wr, 385 op/s; 459 KiB/s, 8 objects/s recovering 2024-09-20T23:54:07.791 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:07 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:54:07.791 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:07 smithi144 ceph-mon[51829]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T23:54:07.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:07 smithi137 ceph-mon[65321]: pgmap v221: 97 pgs: 97 active+clean; 332 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 6.9 KiB/s rd, 4.3 MiB/s wr, 385 op/s; 459 KiB/s, 8 objects/s recovering 2024-09-20T23:54:07.827 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:07 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:54:07.828 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:07 smithi137 ceph-mon[65321]: osdmap e133: 8 total, 7 up, 8 in 2024-09-20T23:54:08.186 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:07 smithi144 podman[58506]: 2024-09-20 23:54:07.789844442 +0000 UTC m=+0.556333019 container remove 946b8c40c3ac593e3bbe230f4bdaa20d8fcfda9f5ed8070097ccf4f7849a0f00 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, distribution-scope=public, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, release=754, vendor=Red Hat, Inc., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-20T23:54:08.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 241314 9.67 MB/sec execute 472 sec latency 1073.635 ms 2024-09-20T23:54:08.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.086333715 +0000 UTC m=+0.020118494 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:08.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.199416839 +0000 UTC m=+0.133201618 container create b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T23:54:08.591 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:08 smithi137 ceph-mon[65321]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T23:54:08.592 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:08.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:08 smithi144 ceph-mon[51829]: osdmap e134: 8 total, 7 up, 8 in 2024-09-20T23:54:08.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:08.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.462083878 +0000 UTC m=+0.395868656 container init b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T23:54:08.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.46674125 +0000 UTC m=+0.400526038 container start b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:54:08.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.56908988 +0000 UTC m=+0.502874671 container attach b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-09-20T23:54:08.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:08 smithi144 podman[58593]: 2024-09-20 23:54:08.689605983 +0000 UTC m=+0.623390771 container died b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-20T23:54:09.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 241654 9.68 MB/sec execute 473 sec latency 36.901 ms 2024-09-20T23:54:09.611 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:09 smithi137 ceph-mon[65321]: pgmap v224: 97 pgs: 1 active+undersized, 10 peering, 5 stale+active+clean, 4 active+undersized+degraded, 77 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.2 MiB/s wr, 566 op/s; 30/1620 objects degraded (1.852%) 2024-09-20T23:54:09.612 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:09 smithi137 ceph-mon[65321]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T23:54:09.612 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:09 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 30/1620 objects degraded (1.852%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:09.906 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:09 smithi144 ceph-mon[51829]: pgmap v224: 97 pgs: 1 active+undersized, 10 peering, 5 stale+active+clean, 4 active+undersized+degraded, 77 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.2 MiB/s wr, 566 op/s; 30/1620 objects degraded (1.852%) 2024-09-20T23:54:09.906 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:09 smithi144 ceph-mon[51829]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-09-20T23:54:09.906 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:09 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 30/1620 objects degraded (1.852%), 4 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:09.907 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 podman[58616]: 2024-09-20 23:54:09.449732512 +0000 UTC m=+0.752117456 container remove b62342ec4ed6f2b039a7beabed09e8e8adf0cb0d9bad3573f1a73dae9a62d928 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:54:09.907 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.5.service: Deactivated successfully. 2024-09-20T23:54:09.907 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 systemd[1]: Stopped Ceph osd.5 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:54:09.907 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.5.service: Consumed 4min 47.902s CPU time. 2024-09-20T23:54:09.907 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 systemd[1]: Starting Ceph osd.5 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:54:10.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 podman[58715]: 2024-09-20 23:54:09.808209127 +0000 UTC m=+0.019197422 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:10.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:09 smithi144 podman[58715]: 2024-09-20 23:54:09.925707007 +0000 UTC m=+0.136695286 container create a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T23:54:10.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 241987 9.67 MB/sec execute 474 sec latency 34.762 ms 2024-09-20T23:54:10.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 podman[58715]: 2024-09-20 23:54:10.19548717 +0000 UTC m=+0.406475459 container init a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 podman[58715]: 2024-09-20 23:54:10.199006919 +0000 UTC m=+0.409995206 container start a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 podman[58715]: 2024-09-20 23:54:10.295732934 +0000 UTC m=+0.506721229 container attach a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 bash[58715]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:10.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:10 smithi144 bash[58715]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:11.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 242259 9.66 MB/sec execute 475 sec latency 37.595 ms 2024-09-20T23:54:11.677 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:11 smithi137 ceph-mon[65321]: pgmap v225: 97 pgs: 4 active+undersized, 10 peering, 13 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.8 MiB/s wr, 327 op/s; 86/1614 objects degraded (5.328%) 2024-09-20T23:54:11.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-mon[51829]: pgmap v225: 97 pgs: 4 active+undersized, 10 peering, 13 active+undersized+degraded, 70 active+clean; 337 MiB data, 3.1 GiB used, 712 GiB / 715 GiB avail; 7.5 KiB/s rd, 3.8 MiB/s wr, 327 op/s; 86/1614 objects degraded (5.328%) 2024-09-20T23:54:11.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T23:54:11.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T23:54:11.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T23:54:11.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-5 --no-mon-config --dev /dev/mapper/vg_nvme-lv_3 2024-09-20T23:54:12.067 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T23:54:12.067 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_3 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_3 /var/lib/ceph/osd/ceph-5/block 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate[58729]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 bash[58715]: --> ceph-volume raw activate successful for osd ID: 5 2024-09-20T23:54:12.068 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:11 smithi144 podman[58715]: 2024-09-20 23:54:11.755432994 +0000 UTC m=+1.966421290 container died a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0) 2024-09-20T23:54:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 242541 9.65 MB/sec execute 476 sec latency 64.441 ms 2024-09-20T23:54:12.705 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:12 smithi137 ceph-mon[65321]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T23:54:12.848 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:12 smithi144 ceph-mon[51829]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-09-20T23:54:12.849 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:12 smithi144 podman[58871]: 2024-09-20 23:54:12.476560755 +0000 UTC m=+0.709606288 container remove a32e362741e33a40bc1a5608f37fbfa0eb1b30c8d10fb5a7c0675edcddc176f3 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70) 2024-09-20T23:54:12.849 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:12 smithi144 podman[58932]: 2024-09-20 23:54:12.594807673 +0000 UTC m=+0.017596375 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:12.849 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:12 smithi144 podman[58932]: 2024-09-20 23:54:12.719268868 +0000 UTC m=+0.142057565 container create a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T23:54:13.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:12 smithi144 podman[58932]: 2024-09-20 23:54:12.972597461 +0000 UTC m=+0.395386174 container init a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-09-20T23:54:13.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:12 smithi144 podman[58932]: 2024-09-20 23:54:12.977321421 +0000 UTC m=+0.400110122 container start a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T23:54:13.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:13 smithi144 ceph-osd[58951]: -- 172.21.15.144:0/71373202 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557ec8f781e0 con 0x557ec8f56000 2024-09-20T23:54:13.194 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:13 smithi144 bash[58932]: a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 2024-09-20T23:54:13.194 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:13 smithi144 systemd[1]: Started Ceph osd.5 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:54:13.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 242851 9.65 MB/sec execute 477 sec latency 70.647 ms 2024-09-20T23:54:13.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:13 smithi137 ceph-mon[65321]: pgmap v226: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.9 MiB/s wr, 398 op/s; 176/1656 objects degraded (10.628%) 2024-09-20T23:54:13.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:13.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:13.736 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:13 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:13.909 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:13 smithi144 ceph-mon[51829]: pgmap v226: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 2.5 KiB/s rd, 3.9 MiB/s wr, 398 op/s; 176/1656 objects degraded (10.628%) 2024-09-20T23:54:13.909 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:13.909 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:13.909 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:13 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:14.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 243178 9.65 MB/sec execute 478 sec latency 61.663 ms 2024-09-20T23:54:14.751 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:14 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 183/1698 objects degraded (10.777%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:15.069 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:14 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 183/1698 objects degraded (10.777%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:15.070 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:14 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:54:14.853+0000 7f8058c29640 -1 Falling back to public interface 2024-09-20T23:54:15.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 243630 9.65 MB/sec execute 479 sec latency 46.934 ms 2024-09-20T23:54:15.779 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:15 smithi137 ceph-mon[65321]: pgmap v227: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 555 op/s; 183/1698 objects degraded (10.777%) 2024-09-20T23:54:16.053 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:15 smithi144 ceph-mon[51829]: pgmap v227: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 5.2 MiB/s wr, 555 op/s; 183/1698 objects degraded (10.777%) 2024-09-20T23:54:16.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 243928 9.64 MB/sec execute 480 sec latency 45.492 ms 2024-09-20T23:54:17.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 244295 9.64 MB/sec execute 481 sec latency 35.066 ms 2024-09-20T23:54:17.301 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:17 smithi137 ceph-mon[65321]: pgmap v228: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.7 MiB/s wr, 480 op/s; 183/1686 objects degraded (10.854%) 2024-09-20T23:54:17.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:17 smithi144 ceph-mon[51829]: pgmap v228: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.7 MiB/s wr, 480 op/s; 183/1686 objects degraded (10.854%) 2024-09-20T23:54:18.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 244645 9.65 MB/sec execute 482 sec latency 39.046 ms 2024-09-20T23:54:18.567 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.568 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.568 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.568 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.568 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.568 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:18.611 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.611 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.612 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.612 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.612 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:18.612 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:19.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 245010 9.66 MB/sec execute 483 sec latency 37.024 ms 2024-09-20T23:54:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:19 smithi144 ceph-mon[51829]: pgmap v229: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s; 181/1623 objects degraded (11.152%) 2024-09-20T23:54:19.611 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:19 smithi137 ceph-mon[65321]: pgmap v229: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 332 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 399 op/s; 181/1623 objects degraded (11.152%) 2024-09-20T23:54:19.924 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:54:20.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 245292 9.66 MB/sec execute 484 sec latency 51.749 ms 2024-09-20T23:54:20.378 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:20 smithi137 ceph-mon[65321]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:20.656 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:20 smithi144 ceph-mon[51829]: from='client.34364 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (7m) 78s ago 16m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (4m) 78s ago 16m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (4m) 3s ago 16m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (7m) 78s ago 16m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (10m) 78s ago 10m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (10m) 3s ago 10m 14.9M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (6m) 78s ago 17m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (5m) 3s ago 15m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (5m) 78s ago 18m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (5m) 3s ago 15m 42.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:54:20.960 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (10m) 78s ago 10m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (10m) 3s ago 10m 86.3M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 78s ago 16m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (6m) 3s ago 15m 16.8M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (3m) 78s ago 15m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (3m) 78s ago 14m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (2m) 78s ago 14m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (86s) 78s ago 13m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (44s) 3s ago 13m 274M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (8s) 3s ago 12m 12.0M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (12m) 3s ago 12m 493M 2704M 17.2.0 e1d6a67b021e 9e2e2cc07bb3 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (12m) 3s ago 12m 768M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:54:20.961 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (6m) 78s ago 15m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:54:21.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 245526 9.65 MB/sec execute 485 sec latency 54.333 ms 2024-09-20T23:54:21.398 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:21 smithi137 ceph-mon[65321]: pgmap v230: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.3 MiB/s wr, 449 op/s; 176/1659 objects degraded (10.609%) 2024-09-20T23:54:21.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:21 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 176/1659 objects degraded (10.609%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:21.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:21 smithi137 ceph-mon[65321]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.404 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:21 smithi137 ceph-mon[65321]: from='client.44307 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.547 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 5 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:54:21.548 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 4, 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 9 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:54:21.549 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:54:21.686 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:21 smithi144 ceph-mon[51829]: pgmap v230: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 331 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.3 MiB/s wr, 449 op/s; 176/1659 objects degraded (10.609%) 2024-09-20T23:54:21.686 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:21 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 176/1659 objects degraded (10.609%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:21.687 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:21 smithi144 ceph-mon[51829]: from='client.34368 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:21.687 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:21 smithi144 ceph-mon[51829]: from='client.44307 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:22.030 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "12/23 daemons upgraded", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:54:22.031 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:54:22.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 245723 9.63 MB/sec execute 486 sec latency 53.929 ms 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3868155439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:22.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:22 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:22.615 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 176/1659 objects degraded (10.609%), 23 pgs degraded 2024-09-20T23:54:22.615 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:54:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: osd.5 (root=default,host=smithi144) is down 2024-09-20T23:54:22.615 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 176/1659 objects degraded (10.609%), 23 pgs degraded 2024-09-20T23:54:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.2 is active+undersized+degraded, acting [1,6] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.3 is active+undersized+degraded, acting [2,7] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.b is active+undersized+degraded, acting [7,4] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.c is active+undersized+degraded, acting [2,0] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.12 is active+undersized+degraded, acting [3,7] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.14 is active+undersized+degraded, acting [6,3] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.15 is active+undersized+degraded, acting [1,0] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.16 is active+undersized+degraded, acting [6,2] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.17 is active+undersized+degraded, acting [6,2] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.18 is active+undersized+degraded, acting [4,7] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1c is active+undersized+degraded, acting [4,2] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1e is active+undersized+degraded, acting [3,0] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.2 is active+undersized+degraded, acting [3,6] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.4 is active+undersized+degraded, acting [1,2] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.5 is active+undersized+degraded, acting [3,2] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.c is active+undersized+degraded, acting [3,6] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.d is active+undersized+degraded, acting [7,6] 2024-09-20T23:54:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.10 is active+undersized+degraded, acting [6,0] 2024-09-20T23:54:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.16 is active+undersized+degraded, acting [7,1] 2024-09-20T23:54:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.17 is active+undersized+degraded, acting [0,3] 2024-09-20T23:54:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1c is active+undersized+degraded, acting [4,1] 2024-09-20T23:54:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1d is active+undersized+degraded, acting [4,6] 2024-09-20T23:54:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1f is active+undersized+degraded, acting [0,2] 2024-09-20T23:54:22.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3868155439' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:22.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:22 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:23.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 245933 9.63 MB/sec execute 487 sec latency 56.206 ms 2024-09-20T23:54:23.367 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:54:23.086+0000 7f8058c29640 -1 osd.5 0 read_superblock omap replica is missing. 2024-09-20T23:54:23.464 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:23 smithi137 ceph-mon[65321]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:23.465 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:23 smithi137 ceph-mon[65321]: pgmap v231: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.9 MiB/s wr, 471 op/s; 187/1671 objects degraded (11.191%) 2024-09-20T23:54:23.465 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:23 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:23.465 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:23 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:54:23.465 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:23 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/901959040' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:54:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-mon[51829]: from='client.34380 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-mon[51829]: pgmap v231: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 336 MiB data, 3.2 GiB used, 712 GiB / 715 GiB avail; 6.2 KiB/s rd, 4.9 MiB/s wr, 471 op/s; 187/1671 objects degraded (11.191%) 2024-09-20T23:54:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:54:23.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:23 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/901959040' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:54:24.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246138 9.61 MB/sec execute 488 sec latency 66.690 ms 2024-09-20T23:54:24.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:24 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:54:24.415+0000 7f8058c29640 -1 osd.5 132 log_to_monitors true 2024-09-20T23:54:25.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246384 9.61 MB/sec execute 489 sec latency 50.746 ms 2024-09-20T23:54:25.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:25 smithi137 ceph-mon[65321]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 504 op/s; 182/1620 objects degraded (11.235%) 2024-09-20T23:54:25.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:25 smithi137 ceph-mon[65321]: from='osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:54:25.532 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:25 smithi137 ceph-mon[65321]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:54:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:25 smithi144 ceph-mon[51829]: pgmap v232: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 341 MiB data, 3.3 GiB used, 712 GiB / 715 GiB avail; 9.0 KiB/s rd, 5.0 MiB/s wr, 504 op/s; 182/1620 objects degraded (11.235%) 2024-09-20T23:54:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:25 smithi144 ceph-mon[51829]: from='osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:54:25.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:25 smithi144 ceph-mon[51829]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-20T23:54:26.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:54:25 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:54:25.884+0000 7f804c9cc640 -1 osd.5 132 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:54:26.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246571 9.60 MB/sec execute 490 sec latency 205.947 ms 2024-09-20T23:54:26.552 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T23:54:26.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: osdmap e135: 8 total, 7 up, 8 in 2024-09-20T23:54:26.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: from='osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:54:26.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:54:26.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: from='osd.5 ' entity='osd.5' 2024-09-20T23:54:26.553 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:26 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 182/1620 objects degraded (11.235%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-20T23:54:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: osdmap e135: 8 total, 7 up, 8 in 2024-09-20T23:54:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: from='osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:54:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:54:26.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: from='osd.5 ' entity='osd.5' 2024-09-20T23:54:26.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:26 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 182/1620 objects degraded (11.235%), 23 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:27.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246788 9.59 MB/sec execute 491 sec latency 361.458 ms 2024-09-20T23:54:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:27 smithi137 ceph-mon[65321]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 455 op/s; 173/1587 objects degraded (10.901%) 2024-09-20T23:54:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:27 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:54:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:27 smithi137 ceph-mon[65321]: osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614] boot 2024-09-20T23:54:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:27 smithi137 ceph-mon[65321]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T23:54:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:27 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:54:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:27 smithi144 ceph-mon[51829]: pgmap v234: 97 pgs: 8 active+undersized, 23 active+undersized+degraded, 66 active+clean; 339 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.6 MiB/s wr, 455 op/s; 173/1587 objects degraded (10.901%) 2024-09-20T23:54:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:27 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:54:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:27 smithi144 ceph-mon[51829]: osd.5 [v2:172.21.15.144:6808/1535078614,v1:172.21.15.144:6809/1535078614] boot 2024-09-20T23:54:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:27 smithi144 ceph-mon[51829]: osdmap e136: 8 total, 8 up, 8 in 2024-09-20T23:54:27.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:27 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-20T23:54:28.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246820 9.57 MB/sec execute 492 sec latency 1333.696 ms 2024-09-20T23:54:29.158 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:28 smithi137 ceph-mon[65321]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T23:54:29.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:28 smithi144 ceph-mon[51829]: osdmap e137: 8 total, 8 up, 8 in 2024-09-20T23:54:29.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 246944 9.56 MB/sec execute 493 sec latency 1997.053 ms 2024-09-20T23:54:29.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:29 smithi137 ceph-mon[65321]: pgmap v237: 97 pgs: 2 remapped+peering, 7 active+undersized, 20 active+undersized+degraded, 68 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 317 op/s; 140/1539 objects degraded (9.097%) 2024-09-20T23:54:29.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:29 smithi137 ceph-mon[65321]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T23:54:30.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:29 smithi144 ceph-mon[51829]: pgmap v237: 97 pgs: 2 remapped+peering, 7 active+undersized, 20 active+undersized+degraded, 68 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.8 KiB/s rd, 3.0 MiB/s wr, 317 op/s; 140/1539 objects degraded (9.097%) 2024-09-20T23:54:30.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:29 smithi144 ceph-mon[51829]: osdmap e138: 8 total, 8 up, 8 in 2024-09-20T23:54:30.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 247269 9.55 MB/sec execute 494 sec latency 339.654 ms 2024-09-20T23:54:30.952 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:30 smithi137 ceph-mon[65321]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T23:54:31.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:30 smithi144 ceph-mon[51829]: osdmap e139: 8 total, 8 up, 8 in 2024-09-20T23:54:31.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 247438 9.54 MB/sec execute 495 sec latency 1020.405 ms 2024-09-20T23:54:31.982 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:31 smithi137 ceph-mon[65321]: pgmap v240: 97 pgs: 2 remapped+peering, 6 active+undersized, 14 active+undersized+degraded, 75 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.3 MiB/s wr, 291 op/s; 107/1518 objects degraded (7.049%); 27 KiB/s, 7 objects/s recovering 2024-09-20T23:54:31.983 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:31 smithi137 ceph-mon[65321]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T23:54:31.983 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:31 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 107/1518 objects degraded (7.049%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:32.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:31 smithi144 ceph-mon[51829]: pgmap v240: 97 pgs: 2 remapped+peering, 6 active+undersized, 14 active+undersized+degraded, 75 active+clean; 337 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 2.3 MiB/s wr, 291 op/s; 107/1518 objects degraded (7.049%); 27 KiB/s, 7 objects/s recovering 2024-09-20T23:54:32.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:31 smithi144 ceph-mon[51829]: osdmap e140: 8 total, 8 up, 8 in 2024-09-20T23:54:32.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:31 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 107/1518 objects degraded (7.049%), 14 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:32.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 247681 9.52 MB/sec execute 496 sec latency 43.133 ms 2024-09-20T23:54:33.015 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:32 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 107/1518 objects degraded (7.049%), 14 pgs degraded) 2024-09-20T23:54:33.016 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:32 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:54:33.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 247999 9.54 MB/sec execute 497 sec latency 54.361 ms 2024-09-20T23:54:33.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:32 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 107/1518 objects degraded (7.049%), 14 pgs degraded) 2024-09-20T23:54:33.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:32 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:54:34.036 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:33 smithi137 ceph-mon[65321]: pgmap v242: 97 pgs: 5 peering, 92 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 286 op/s; 596 KiB/s, 48 objects/s recovering 2024-09-20T23:54:34.036 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:34.036 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:34.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 248382 9.54 MB/sec execute 498 sec latency 54.156 ms 2024-09-20T23:54:34.395 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:33 smithi144 ceph-mon[51829]: pgmap v242: 97 pgs: 5 peering, 92 active+clean; 333 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.8 KiB/s rd, 2.2 MiB/s wr, 286 op/s; 596 KiB/s, 48 objects/s recovering 2024-09-20T23:54:34.396 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:34.396 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:35.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 248672 9.55 MB/sec execute 499 sec latency 96.833 ms 2024-09-20T23:54:36.100 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:35 smithi137 ceph-mon[65321]: pgmap v243: 97 pgs: 5 peering, 92 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.4 MiB/s wr, 465 op/s; 431 KiB/s, 34 objects/s recovering 2024-09-20T23:54:36.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 248893 9.54 MB/sec execute 500 sec latency 57.823 ms 2024-09-20T23:54:36.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:35 smithi144 ceph-mon[51829]: pgmap v243: 97 pgs: 5 peering, 92 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 6.8 KiB/s rd, 3.4 MiB/s wr, 465 op/s; 431 KiB/s, 34 objects/s recovering 2024-09-20T23:54:37.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 249072 9.53 MB/sec execute 501 sec latency 54.201 ms 2024-09-20T23:54:38.183 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:37 smithi137 ceph-mon[65321]: pgmap v244: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 410 op/s; 672 KiB/s, 32 objects/s recovering 2024-09-20T23:54:38.183 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:38.278 INFO:teuthology.orchestra.run.smithi137.stdout: 5 249265 9.52 MB/sec execute 502 sec latency 60.462 ms 2024-09-20T23:54:38.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:37 smithi144 ceph-mon[51829]: pgmap v244: 97 pgs: 97 active+clean; 338 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 3.6 MiB/s wr, 410 op/s; 672 KiB/s, 32 objects/s recovering 2024-09-20T23:54:38.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:39.214 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:38 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:39.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:39 smithi137 ceph-mon[65321]: Upgrade: osd.6 is safe to restart 2024-09-20T23:54:39.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:39.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:54:39.215 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:39.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 249524 9.51 MB/sec execute 503 sec latency 47.112 ms 2024-09-20T23:54:39.325 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:39 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["6"], "max": 16}]: dispatch 2024-09-20T23:54:39.325 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:39 smithi144 ceph-mon[51829]: Upgrade: osd.6 is safe to restart 2024-09-20T23:54:39.325 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:39.325 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-20T23:54:39.325 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:40.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:40 smithi137 ceph-mon[65321]: pgmap v245: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.4 MiB/s wr, 287 op/s; 579 KiB/s, 24 objects/s recovering 2024-09-20T23:54:40.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:40 smithi137 ceph-mon[65321]: Upgrade: Updating osd.6 2024-09-20T23:54:40.232 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:40 smithi137 ceph-mon[65321]: Deploying daemon osd.6 on smithi144 2024-09-20T23:54:40.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 249782 9.51 MB/sec execute 504 sec latency 43.843 ms 2024-09-20T23:54:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:40 smithi144 ceph-mon[51829]: pgmap v245: 97 pgs: 97 active+clean; 335 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 2.6 KiB/s rd, 3.4 MiB/s wr, 287 op/s; 579 KiB/s, 24 objects/s recovering 2024-09-20T23:54:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:40 smithi144 ceph-mon[51829]: Upgrade: Updating osd.6 2024-09-20T23:54:40.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:40 smithi144 ceph-mon[51829]: Deploying daemon osd.6 on smithi144 2024-09-20T23:54:41.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 249994 9.50 MB/sec execute 505 sec latency 52.064 ms 2024-09-20T23:54:42.020 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:42 smithi137 ceph-mon[65321]: pgmap v246: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.7 MiB/s wr, 375 op/s; 482 KiB/s, 20 objects/s recovering 2024-09-20T23:54:42.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:42 smithi144 ceph-mon[51829]: pgmap v246: 97 pgs: 97 active+clean; 341 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.7 MiB/s wr, 375 op/s; 482 KiB/s, 20 objects/s recovering 2024-09-20T23:54:42.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 250261 9.50 MB/sec execute 506 sec latency 79.649 ms 2024-09-20T23:54:42.694 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:42 smithi144 systemd[1]: Stopping Ceph osd.6 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:54:42.694 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:42 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[36161]: 2024-09-20T23:54:42.530+0000 7fb73a5a5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:54:42.694 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:42 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[36161]: 2024-09-20T23:54:42.530+0000 7fb73a5a5700 -1 osd.6 140 *** Got signal Terminated *** 2024-09-20T23:54:42.694 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:42 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[36161]: 2024-09-20T23:54:42.530+0000 7fb73a5a5700 -1 osd.6 140 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:54:43.055 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:43 smithi137 ceph-mon[65321]: osd.6 marked itself down and dead 2024-09-20T23:54:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 250533 9.49 MB/sec execute 507 sec latency 130.871 ms 2024-09-20T23:54:43.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:43 smithi144 ceph-mon[51829]: osd.6 marked itself down and dead 2024-09-20T23:54:43.943 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:43 smithi144 podman[60757]: 2024-09-20 23:54:43.681930287 +0000 UTC m=+1.251246454 container died 9e2e2cc07bb3d36050ada733fdc5c211372ac29408dc5b933b49c804ba80a5f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , name=centos-stream, release=754, GIT_CLEAN=True, RELEASE=HEAD, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.component=centos-stream-container, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-20T23:54:44.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 250678 9.47 MB/sec execute 508 sec latency 943.672 ms 2024-09-20T23:54:44.297 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:44 smithi137 ceph-mon[65321]: pgmap v247: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.9 MiB/s wr, 351 op/s; 351 KiB/s, 6 objects/s recovering 2024-09-20T23:54:44.297 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:44 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:54:44.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:44 smithi137 ceph-mon[65321]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T23:54:44.298 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:44.442 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:44 smithi144 ceph-mon[51829]: pgmap v247: 97 pgs: 97 active+clean; 343 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.9 MiB/s wr, 351 op/s; 351 KiB/s, 6 objects/s recovering 2024-09-20T23:54:44.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:44 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:54:44.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:44 smithi144 ceph-mon[51829]: osdmap e141: 8 total, 7 up, 8 in 2024-09-20T23:54:44.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:44.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:44 smithi144 podman[60757]: 2024-09-20 23:54:44.156226241 +0000 UTC m=+1.725542443 container cleanup 9e2e2cc07bb3d36050ada733fdc5c211372ac29408dc5b933b49c804ba80a5f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, RELEASE=HEAD, com.redhat.component=centos-stream-container, name=centos-stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, release=754, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-09-20T23:54:44.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:44 smithi144 bash[60757]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6 2024-09-20T23:54:44.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:44 smithi144 podman[60777]: 2024-09-20 23:54:44.441335827 +0000 UTC m=+0.750493272 container remove 9e2e2cc07bb3d36050ada733fdc5c211372ac29408dc5b933b49c804ba80a5f3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, ceph=True, build-date=2022-05-03T08:36:31.336870, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, GIT_CLEAN=True, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8) 2024-09-20T23:54:45.073 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:44 smithi144 podman[60867]: 2024-09-20 23:54:44.683122138 +0000 UTC m=+0.016491956 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:45.073 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:45 smithi144 podman[60867]: 2024-09-20 23:54:45.017419559 +0000 UTC m=+0.350789374 container create af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-20T23:54:45.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 251003 9.46 MB/sec execute 509 sec latency 72.656 ms 2024-09-20T23:54:45.318 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:45 smithi137 ceph-mon[65321]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T23:54:45.318 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:45 smithi137 ceph-mon[65321]: pgmap v250: 97 pgs: 11 peering, 5 stale+active+clean, 5 active+undersized+degraded, 76 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.1 MiB/s wr, 421 op/s; 37/1647 objects degraded (2.247%) 2024-09-20T23:54:45.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:45 smithi144 ceph-mon[51829]: osdmap e142: 8 total, 7 up, 8 in 2024-09-20T23:54:45.406 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:45 smithi144 ceph-mon[51829]: pgmap v250: 97 pgs: 11 peering, 5 stale+active+clean, 5 active+undersized+degraded, 76 active+clean; 336 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 3.1 KiB/s rd, 4.1 MiB/s wr, 421 op/s; 37/1647 objects degraded (2.247%) 2024-09-20T23:54:45.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:45 smithi144 podman[60867]: 2024-09-20 23:54:45.405138188 +0000 UTC m=+0.738508021 container init af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T23:54:45.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:45 smithi144 podman[60867]: 2024-09-20 23:54:45.408646186 +0000 UTC m=+0.742016011 container start af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-20T23:54:45.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:45 smithi144 podman[60867]: 2024-09-20 23:54:45.506344099 +0000 UTC m=+0.839713925 container attach af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True) 2024-09-20T23:54:45.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:45 smithi144 podman[60867]: 2024-09-20 23:54:45.599437711 +0000 UTC m=+0.932807533 container died af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD) 2024-09-20T23:54:46.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 251191 9.45 MB/sec execute 510 sec latency 49.574 ms 2024-09-20T23:54:46.326 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:46 smithi137 ceph-mon[65321]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-20T23:54:46.327 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:46 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 37/1647 objects degraded (2.247%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:46.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:46 smithi144 ceph-mon[51829]: Health check failed: Reduced data availability: 2 pgs peering (PG_AVAILABILITY) 2024-09-20T23:54:46.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:46 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 37/1647 objects degraded (2.247%), 5 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:46.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 podman[60890]: 2024-09-20 23:54:46.223246086 +0000 UTC m=+0.617170823 container remove af01088c79382a8402cc729d5e42af3b8bb6a42586e4562f179bc3acac75b968 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-09-20T23:54:46.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.6.service: Deactivated successfully. 2024-09-20T23:54:46.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 systemd[1]: Stopped Ceph osd.6 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:54:46.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.6.service: Consumed 4min 17.453s CPU time. 2024-09-20T23:54:46.804 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 systemd[1]: Starting Ceph osd.6 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:54:46.804 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 podman[60989]: 2024-09-20 23:54:46.562395251 +0000 UTC m=+0.017441736 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:46.804 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 podman[60989]: 2024-09-20 23:54:46.677281551 +0000 UTC m=+0.132328033 container create a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, ceph=True) 2024-09-20T23:54:47.082 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 podman[60989]: 2024-09-20 23:54:46.939370903 +0000 UTC m=+0.394417396 container init a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-20T23:54:47.082 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:46 smithi144 podman[60989]: 2024-09-20 23:54:46.942885103 +0000 UTC m=+0.397931594 container start a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-09-20T23:54:47.082 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 podman[60989]: 2024-09-20 23:54:47.039184454 +0000 UTC m=+0.494230937 container attach a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:54:47.098 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:47 smithi137 ceph-mon[65321]: pgmap v251: 97 pgs: 2 active+undersized, 11 peering, 2 stale+active+clean, 11 active+undersized+degraded, 71 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 426 op/s; 82/1575 objects degraded (5.206%) 2024-09-20T23:54:47.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 251488 9.46 MB/sec execute 511 sec latency 64.393 ms 2024-09-20T23:54:47.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:47 smithi144 ceph-mon[51829]: pgmap v251: 97 pgs: 2 active+undersized, 11 peering, 2 stale+active+clean, 11 active+undersized+degraded, 71 active+clean; 330 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 8.1 KiB/s rd, 3.6 MiB/s wr, 426 op/s; 82/1575 objects degraded (5.206%) 2024-09-20T23:54:47.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:47.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 bash[60989]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:47.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:47.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 bash[60989]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:54:48.150 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T23:54:48.150 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 bash[60989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T23:54:48.150 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T23:54:48.150 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:47 smithi144 bash[60989]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-6 --no-mon-config --dev /dev/mapper/vg_nvme-lv_2 2024-09-20T23:54:48.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 251927 9.47 MB/sec execute 512 sec latency 48.097 ms 2024-09-20T23:54:48.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 bash[60989]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_2 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 bash[60989]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 bash[60989]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_2 /var/lib/ceph/osd/ceph-6/block 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 bash[60989]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate[61004]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 bash[60989]: --> ceph-volume raw activate successful for osd ID: 6 2024-09-20T23:54:48.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 podman[60989]: 2024-09-20 23:54:48.183062813 +0000 UTC m=+1.638109305 container died a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T23:54:49.198 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:48 smithi144 podman[61145]: 2024-09-20 23:54:48.946601398 +0000 UTC m=+0.757642492 container remove a1fa2317d8c5e3354ece51d45f696ecb68027c7c1fd983771dcb22e5c5d082cc (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-activate, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2) 2024-09-20T23:54:49.199 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 podman[61205]: 2024-09-20 23:54:49.069521609 +0000 UTC m=+0.017743065 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:54:49.199 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 podman[61205]: 2024-09-20 23:54:49.197766586 +0000 UTC m=+0.145988078 container create c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=) 2024-09-20T23:54:49.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 252305 9.48 MB/sec execute 513 sec latency 55.058 ms 2024-09-20T23:54:49.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:49.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:49.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:49 smithi137 ceph-mon[65321]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 327 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.1 MiB/s wr, 308 op/s; 182/1548 objects degraded (11.757%) 2024-09-20T23:54:49.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:49.472 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:54:49.473 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:49 smithi144 ceph-mon[51829]: pgmap v252: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 327 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.1 MiB/s wr, 308 op/s; 182/1548 objects degraded (11.757%) 2024-09-20T23:54:49.473 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 podman[61205]: 2024-09-20 23:54:49.442291125 +0000 UTC m=+0.390512565 container init c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-09-20T23:54:49.473 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 podman[61205]: 2024-09-20 23:54:49.447443919 +0000 UTC m=+0.395665360 container start c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2) 2024-09-20T23:54:49.943 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 ceph-osd[61223]: -- 172.21.15.144:0/2081179035 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5562143721e0 con 0x55621355fc00 2024-09-20T23:54:49.944 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 bash[61205]: c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 2024-09-20T23:54:49.944 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:49 smithi144 systemd[1]: Started Ceph osd.6 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:54:50.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 252525 9.48 MB/sec execute 514 sec latency 69.706 ms 2024-09-20T23:54:50.408 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:50 smithi137 ceph-mon[65321]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-20T23:54:50.408 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:50.408 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:50.408 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:50 smithi144 ceph-mon[51829]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs peering) 2024-09-20T23:54:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:50.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:50.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:51.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 252758 9.47 MB/sec execute 515 sec latency 68.151 ms 2024-09-20T23:54:51.371 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:51 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:54:51.076+0000 7f569f74b640 -1 Falling back to public interface 2024-09-20T23:54:51.371 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:51 smithi144 ceph-mon[51829]: pgmap v253: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.0 MiB/s wr, 387 op/s; 181/1542 objects degraded (11.738%) 2024-09-20T23:54:51.371 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:51 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 182/1548 objects degraded (11.757%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:51.424 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:51 smithi137 ceph-mon[65321]: pgmap v253: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 331 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 4.7 KiB/s rd, 3.0 MiB/s wr, 387 op/s; 181/1542 objects degraded (11.738%) 2024-09-20T23:54:51.425 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:51 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 182/1548 objects degraded (11.757%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:52.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 253056 9.46 MB/sec execute 516 sec latency 67.415 ms 2024-09-20T23:54:53.163 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:54:53.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 253323 9.45 MB/sec execute 517 sec latency 62.297 ms 2024-09-20T23:54:53.454 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:53 smithi137 ceph-mon[65321]: pgmap v254: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 410 op/s; 193/1641 objects degraded (11.761%) 2024-09-20T23:54:53.631 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:53 smithi144 ceph-mon[51829]: pgmap v254: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.7 KiB/s rd, 4.0 MiB/s wr, 410 op/s; 193/1641 objects degraded (11.761%) 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (8m) 111s ago 17m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (4m) 111s ago 17m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (4m) 0s ago 16m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (8m) 111s ago 16m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (11m) 111s ago 11m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (11m) 0s ago 11m 15.0M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (7m) 111s ago 18m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:54:54.214 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (6m) 0s ago 16m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (6m) 111s ago 18m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (5m) 0s ago 16m 44.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (10m) 111s ago 10m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (10m) 0s ago 10m 86.3M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (7m) 111s ago 16m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (7m) 0s ago 16m 16.5M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (4m) 111s ago 15m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (3m) 111s ago 15m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (2m) 111s ago 14m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (2m) 111s ago 14m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (77s) 0s ago 13m 307M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (41s) 0s ago 13m 254M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (5s) 0s ago 13m 12.1M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (12m) 0s ago 12m 806M 2704M 17.2.0 e1d6a67b021e 9465469824bf 2024-09-20T23:54:54.215 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (6m) 111s ago 16m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:54:54.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 253516 9.44 MB/sec execute 518 sec latency 60.651 ms 2024-09-20T23:54:54.820 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:54:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:54:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:54:54.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 6 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 3, 2024-09-20T23:54:54.821 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 10 2024-09-20T23:54:54.822 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:54:54.822 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:54.967 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:54 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2379593202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:55.185 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='client.34390 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='client.34394 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:55.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:54 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2379593202' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:55.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 253784 9.45 MB/sec execute 519 sec latency 49.017 ms 2024-09-20T23:54:55.347 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:54:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:54:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:54:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:54:55.347 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "13/23 daemons upgraded", 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:54:55.348 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:54:55.993 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:55 smithi137 ceph-mon[65321]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:55.994 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:55 smithi137 ceph-mon[65321]: pgmap v255: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.7 MiB/s wr, 475 op/s; 202/1671 objects degraded (12.089%) 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down; Degraded data redundancy: 202/1671 objects degraded (12.089%), 25 pgs degraded 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout: osd.6 (root=default,host=smithi144) is down 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 202/1671 objects degraded (12.089%), 25 pgs degraded 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout: pg 1.0 is active+undersized+degraded, acting [7,0] 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.2 is active+undersized+degraded, acting [5,1] 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.6 is active+undersized+degraded, acting [1,4] 2024-09-20T23:54:56.054 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.7 is active+undersized+degraded, acting [7,2] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.11 is active+undersized+degraded, acting [4,1] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.14 is active+undersized+degraded, acting [3,5] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.16 is active+undersized+degraded, acting [5,2] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.17 is active+undersized+degraded, acting [5,2] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.19 is active+undersized+degraded, acting [3,0] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1a is active+undersized+degraded, acting [4,7] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1b is active+undersized+degraded, acting [3,7] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1d is active+undersized+degraded, acting [7,0] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.0 is active+undersized+degraded, acting [1,2] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.2 is active+undersized+degraded, acting [3,5] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.3 is active+undersized+degraded, acting [4,0] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.a is active+undersized+degraded, acting [4,1] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.c is active+undersized+degraded, acting [5,3] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.d is active+undersized+degraded, acting [7,5] 2024-09-20T23:54:56.055 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.10 is active+undersized+degraded, acting [0,5] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.11 is active+undersized+degraded, acting [7,4] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.14 is active+undersized+degraded, acting [4,7] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1d is active+undersized+degraded, acting [5,4] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1e is active+undersized+degraded, acting [3,2] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.5 is active+undersized+degraded, acting [4,7] 2024-09-20T23:54:56.056 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.a is active+undersized+degraded, acting [1,7] 2024-09-20T23:54:56.169 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:55 smithi144 ceph-mon[51829]: from='client.44325 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:56.169 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:55 smithi144 ceph-mon[51829]: pgmap v255: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 341 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 6.3 KiB/s rd, 4.7 MiB/s wr, 475 op/s; 202/1671 objects degraded (12.089%) 2024-09-20T23:54:56.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 254073 9.44 MB/sec execute 520 sec latency 94.214 ms 2024-09-20T23:54:57.036 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:56 smithi137 ceph-mon[65321]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:56 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/874441129' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:54:57.037 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:56 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 202/1671 objects degraded (12.089%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:56 smithi144 ceph-mon[51829]: from='client.34406 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:54:57.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:56 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/874441129' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:54:57.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:56 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 202/1671 objects degraded (12.089%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:54:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 254435 9.42 MB/sec execute 521 sec latency 61.374 ms 2024-09-20T23:54:58.087 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:57 smithi137 ceph-mon[65321]: pgmap v256: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.7 MiB/s wr, 318 op/s; 193/1587 objects degraded (12.161%) 2024-09-20T23:54:58.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:57 smithi144 ceph-mon[51829]: pgmap v256: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 338 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 8.9 KiB/s rd, 3.7 MiB/s wr, 318 op/s; 193/1587 objects degraded (12.161%) 2024-09-20T23:54:58.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 254675 9.41 MB/sec execute 522 sec latency 39.245 ms 2024-09-20T23:54:59.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 254880 9.41 MB/sec execute 523 sec latency 75.046 ms 2024-09-20T23:54:59.366 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:54:59.027+0000 7f569f74b640 -1 osd.6 0 read_superblock omap replica is missing. 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 385 op/s; 183/1551 objects degraded (11.799%) 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.395 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:54:59.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:54:59.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:54:59 smithi137 ceph-mon[65321]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:54:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: pgmap v257: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 334 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.2 MiB/s wr, 385 op/s; 183/1551 objects degraded (11.799%) 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:54:59.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:54:59.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:54:59 smithi144 ceph-mon[51829]: Upgrade: unsafe to stop osd(s) at this time (12 PGs are or would become offline) 2024-09-20T23:55:00.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255257 9.42 MB/sec execute 524 sec latency 39.893 ms 2024-09-20T23:55:00.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:00 smithi137 ceph-mon[65321]: from='osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:55:00.421 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:00 smithi137 ceph-mon[65321]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:55:00.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:00 smithi144 ceph-mon[51829]: from='osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:55:00.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:00 smithi144 ceph-mon[51829]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-20T23:55:00.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:55:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:55:00.168+0000 7f569f74b640 -1 osd.6 140 log_to_monitors true 2024-09-20T23:55:00.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:55:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:55:00.417+0000 7f56974f6640 -1 osd.6 140 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:55:01.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255800 9.44 MB/sec execute 525 sec latency 51.531 ms 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: pgmap v258: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 462 op/s; 180/1554 objects degraded (11.583%) 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: from='osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:01.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:01 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 180/1554 objects degraded (11.583%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: pgmap v258: 97 pgs: 10 active+undersized, 25 active+undersized+degraded, 62 active+clean; 337 MiB data, 2.9 GiB used, 712 GiB / 715 GiB avail; 5.6 KiB/s rd, 4.4 MiB/s wr, 462 op/s; 180/1554 objects degraded (11.583%) 2024-09-20T23:55:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-20T23:55:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: from='osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: osdmap e143: 8 total, 7 up, 8 in 2024-09-20T23:55:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:01 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 180/1554 objects degraded (11.583%), 25 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:02.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255845 9.43 MB/sec execute 526 sec latency 898.999 ms 2024-09-20T23:55:02.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:02 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:55:02.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:02 smithi137 ceph-mon[65321]: osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774] boot 2024-09-20T23:55:02.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:02 smithi137 ceph-mon[65321]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T23:55:02.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:02 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:55:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:02 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:55:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:02 smithi144 ceph-mon[51829]: osd.6 [v2:172.21.15.144:6816/3716585774,v1:172.21.15.144:6817/3716585774] boot 2024-09-20T23:55:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:02 smithi144 ceph-mon[51829]: osdmap e144: 8 total, 8 up, 8 in 2024-09-20T23:55:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:02 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-20T23:55:03.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255845 9.41 MB/sec execute 527 sec latency 1899.027 ms 2024-09-20T23:55:03.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:03 smithi137 ceph-mon[65321]: pgmap v261: 97 pgs: 9 peering, 7 active+undersized, 19 active+undersized+degraded, 62 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.6 MiB/s wr, 404 op/s; 124/1575 objects degraded (7.873%) 2024-09-20T23:55:03.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:03 smithi137 ceph-mon[65321]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T23:55:03.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:03.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:03 smithi144 ceph-mon[51829]: pgmap v261: 97 pgs: 9 peering, 7 active+undersized, 19 active+undersized+degraded, 62 active+clean; 336 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 3.6 MiB/s wr, 404 op/s; 124/1575 objects degraded (7.873%) 2024-09-20T23:55:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:03 smithi144 ceph-mon[51829]: osdmap e145: 8 total, 8 up, 8 in 2024-09-20T23:55:03.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:03.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:04.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255941 9.39 MB/sec execute 528 sec latency 2319.662 ms 2024-09-20T23:55:04.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:04 smithi137 ceph-mon[65321]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T23:55:04.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:04 smithi144 ceph-mon[51829]: osdmap e146: 8 total, 8 up, 8 in 2024-09-20T23:55:05.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 255963 9.38 MB/sec execute 529 sec latency 884.781 ms 2024-09-20T23:55:05.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:05 smithi137 ceph-mon[65321]: pgmap v264: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 13 active+undersized+degraded, 71 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.5 MiB/s wr, 179 op/s; 98/1575 objects degraded (6.222%); 76 KiB/s, 7 objects/s recovering 2024-09-20T23:55:05.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:05 smithi137 ceph-mon[65321]: osdmap e147: 8 total, 8 up, 8 in 2024-09-20T23:55:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:05 smithi144 ceph-mon[51829]: pgmap v264: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 9 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized, 13 active+undersized+degraded, 71 active+clean; 333 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 6.7 KiB/s rd, 3.5 MiB/s wr, 179 op/s; 98/1575 objects degraded (6.222%); 76 KiB/s, 7 objects/s recovering 2024-09-20T23:55:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:05 smithi144 ceph-mon[51829]: osdmap e147: 8 total, 8 up, 8 in 2024-09-20T23:55:06.279 INFO:teuthology.orchestra.run.smithi137.stdout: 5 256121 9.37 MB/sec execute 530 sec latency 1046.317 ms 2024-09-20T23:55:06.449 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:06 smithi137 ceph-mon[65321]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T23:55:06.450 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:06 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 98/1575 objects degraded (6.222%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:06.450 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:06 smithi137 ceph-mon[65321]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T23:55:06.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:06 smithi144 ceph-mon[51829]: osdmap e148: 8 total, 8 up, 8 in 2024-09-20T23:55:06.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:06 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 98/1575 objects degraded (6.222%), 15 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:06.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:06 smithi144 ceph-mon[51829]: osdmap e149: 8 total, 8 up, 8 in 2024-09-20T23:55:07.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 256380 9.37 MB/sec execute 531 sec latency 329.834 ms 2024-09-20T23:55:07.459 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:07 smithi137 ceph-mon[65321]: pgmap v267: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 peering, 1 active+recovering+undersized+degraded+remapped, 84 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.2 MiB/s wr, 313 op/s; 27/1614 objects degraded (1.673%); 1.1 MiB/s, 26 objects/s recovering 2024-09-20T23:55:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:07 smithi144 ceph-mon[51829]: pgmap v267: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 11 peering, 1 active+recovering+undersized+degraded+remapped, 84 active+clean; 338 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 3.5 KiB/s rd, 3.2 MiB/s wr, 313 op/s; 27/1614 objects degraded (1.673%); 1.1 MiB/s, 26 objects/s recovering 2024-09-20T23:55:08.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 256687 9.36 MB/sec execute 532 sec latency 71.658 ms 2024-09-20T23:55:09.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 256969 9.35 MB/sec execute 533 sec latency 40.139 ms 2024-09-20T23:55:09.517 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:09 smithi137 ceph-mon[65321]: pgmap v269: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.8 MiB/s wr, 259 op/s; 27/1632 objects degraded (1.654%); 1.2 MiB/s, 46 keys/s, 28 objects/s recovering 2024-09-20T23:55:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:09 smithi144 ceph-mon[51829]: pgmap v269: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 2 peering, 1 active+recovering+undersized+degraded+remapped, 93 active+clean; 339 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.9 KiB/s rd, 2.8 MiB/s wr, 259 op/s; 27/1632 objects degraded (1.654%); 1.2 MiB/s, 46 keys/s, 28 objects/s recovering 2024-09-20T23:55:10.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 257252 9.35 MB/sec execute 534 sec latency 66.739 ms 2024-09-20T23:55:10.556 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:10 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1632 objects degraded (1.654%), 2 pgs degraded) 2024-09-20T23:55:10.557 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:10 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:55:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:10 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 27/1632 objects degraded (1.654%), 2 pgs degraded) 2024-09-20T23:55:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:10 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:55:11.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 257594 9.34 MB/sec execute 535 sec latency 47.224 ms 2024-09-20T23:55:11.583 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:11 smithi137 ceph-mon[65321]: pgmap v270: 97 pgs: 2 peering, 95 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.4 MiB/s wr, 207 op/s; 985 KiB/s, 37 keys/s, 29 objects/s recovering 2024-09-20T23:55:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:11 smithi144 ceph-mon[51829]: pgmap v270: 97 pgs: 2 peering, 95 active+clean; 340 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 2.3 KiB/s rd, 2.4 MiB/s wr, 207 op/s; 985 KiB/s, 37 keys/s, 29 objects/s recovering 2024-09-20T23:55:12.280 INFO:teuthology.orchestra.run.smithi137.stdout: 5 258030 9.34 MB/sec execute 536 sec latency 56.062 ms 2024-09-20T23:55:13.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 258320 9.33 MB/sec execute 537 sec latency 44.836 ms 2024-09-20T23:55:13.732 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:13 smithi137 ceph-mon[65321]: pgmap v271: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 365 op/s; 1.2 MiB/s, 28 keys/s, 22 objects/s recovering 2024-09-20T23:55:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:13 smithi144 ceph-mon[51829]: pgmap v271: 97 pgs: 97 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.6 MiB/s wr, 365 op/s; 1.2 MiB/s, 28 keys/s, 22 objects/s recovering 2024-09-20T23:55:14.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 258698 9.35 MB/sec execute 538 sec latency 33.671 ms 2024-09-20T23:55:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:14 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:55:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:14 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:55:14.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:14 smithi144 ceph-mon[51829]: Upgrade: osd.7 is safe to restart 2024-09-20T23:55:14.759 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:14 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:55:14.760 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:14 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "osd ok-to-stop", "ids": ["7"], "max": 16}]: dispatch 2024-09-20T23:55:14.760 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:14 smithi137 ceph-mon[65321]: Upgrade: osd.7 is safe to restart 2024-09-20T23:55:15.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 259084 9.35 MB/sec execute 539 sec latency 50.368 ms 2024-09-20T23:55:15.779 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:15 smithi137 ceph-mon[65321]: pgmap v272: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 377 op/s; 708 KiB/s, 25 keys/s, 16 objects/s recovering 2024-09-20T23:55:15.780 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:15.780 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:55:15.780 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:15.815 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:15 smithi144 ceph-mon[51829]: pgmap v272: 97 pgs: 97 active+clean; 330 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 3.4 MiB/s wr, 377 op/s; 708 KiB/s, 25 keys/s, 16 objects/s recovering 2024-09-20T23:55:15.815 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:15.815 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-20T23:55:15.815 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:16.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 259458 9.36 MB/sec execute 540 sec latency 46.001 ms 2024-09-20T23:55:16.554 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:16 smithi137 ceph-mon[65321]: Upgrade: Updating osd.7 2024-09-20T23:55:16.555 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:16 smithi137 ceph-mon[65321]: Deploying daemon osd.7 on smithi144 2024-09-20T23:55:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:16 smithi144 ceph-mon[51829]: Upgrade: Updating osd.7 2024-09-20T23:55:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:16 smithi144 ceph-mon[51829]: Deploying daemon osd.7 on smithi144 2024-09-20T23:55:17.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 259664 9.36 MB/sec execute 541 sec latency 69.822 ms 2024-09-20T23:55:17.567 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:17 smithi137 ceph-mon[65321]: pgmap v273: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.0 MiB/s wr, 447 op/s; 596 KiB/s, 22 keys/s, 10 objects/s recovering 2024-09-20T23:55:17.838 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:17 smithi144 ceph-mon[51829]: pgmap v273: 97 pgs: 97 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.0 MiB/s wr, 447 op/s; 596 KiB/s, 22 keys/s, 10 objects/s recovering 2024-09-20T23:55:18.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 259854 9.35 MB/sec execute 542 sec latency 70.314 ms 2024-09-20T23:55:19.087 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:18 smithi144 systemd[1]: Stopping Ceph osd.7 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:55:19.287 INFO:teuthology.orchestra.run.smithi137.stdout: 5 260051 9.34 MB/sec execute 543 sec latency 61.081 ms 2024-09-20T23:55:19.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:19.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:19.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:19 smithi137 ceph-mon[65321]: pgmap v274: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 405 op/s; 504 KiB/s, 19 keys/s, 8 objects/s recovering 2024-09-20T23:55:19.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:19.371 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:19 smithi137 ceph-mon[65321]: osd.7 marked itself down and dead 2024-09-20T23:55:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-mon[51829]: pgmap v274: 97 pgs: 97 active+clean; 338 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 5.7 KiB/s rd, 4.1 MiB/s wr, 405 op/s; 504 KiB/s, 19 keys/s, 8 objects/s recovering 2024-09-20T23:55:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:19.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-mon[51829]: osd.7 marked itself down and dead 2024-09-20T23:55:19.444 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[39206]: 2024-09-20T23:55:19.083+0000 7f987dbab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:55:19.444 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[39206]: 2024-09-20T23:55:19.083+0000 7f987dbab700 -1 osd.7 149 *** Got signal Terminated *** 2024-09-20T23:55:19.444 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[39206]: 2024-09-20T23:55:19.083+0000 7f987dbab700 -1 osd.7 149 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:55:20.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 260224 9.32 MB/sec execute 544 sec latency 861.287 ms 2024-09-20T23:55:20.313 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:20 smithi144 ceph-mon[51829]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:55:20.314 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:20 smithi144 ceph-mon[51829]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T23:55:20.314 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:20 smithi144 ceph-mon[51829]: osdmap e150: 8 total, 7 up, 8 in 2024-09-20T23:55:20.314 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:20 smithi144 podman[63004]: 2024-09-20 23:55:20.028400076 +0000 UTC m=+1.096253722 container died 9465469824bf77b35dfdad9e839de9c811268d5f60f707a79b459e6b4f481af3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, release=754, RELEASE=HEAD, GIT_CLEAN=True, version=8, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-type=git, distribution-scope=public, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, io.openshift.expose-services=, GIT_BRANCH=HEAD) 2024-09-20T23:55:20.485 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:20 smithi137 ceph-mon[65321]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-20T23:55:20.486 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:20 smithi137 ceph-mon[65321]: Health check failed: all OSDs are running squid or later but require_osd_release < squid (OSD_UPGRADE_FINISHED) 2024-09-20T23:55:20.486 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:20 smithi137 ceph-mon[65321]: osdmap e150: 8 total, 7 up, 8 in 2024-09-20T23:55:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:20 smithi144 podman[63004]: 2024-09-20 23:55:20.411594543 +0000 UTC m=+1.479448177 container cleanup 9465469824bf77b35dfdad9e839de9c811268d5f60f707a79b459e6b4f481af3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-09-20T23:55:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:20 smithi144 bash[63004]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7 2024-09-20T23:55:21.089 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:20 smithi144 podman[63051]: 2024-09-20 23:55:20.721985418 +0000 UTC m=+0.689475087 container remove 9465469824bf77b35dfdad9e839de9c811268d5f60f707a79b459e6b4f481af3 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, io.openshift.tags=base centos centos-stream, distribution-scope=public, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-type=git, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vendor=Red Hat, Inc., name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True) 2024-09-20T23:55:21.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 260493 9.32 MB/sec execute 545 sec latency 952.153 ms 2024-09-20T23:55:21.351 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:20.992135135 +0000 UTC m=+0.019200152 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:55:21.352 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:21.098177524 +0000 UTC m=+0.125242530 container create a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:55:21.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:21 smithi137 ceph-mon[65321]: pgmap v276: 97 pgs: 12 stale+active+clean, 85 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 474 op/s; 410 KiB/s, 0 objects/s recovering 2024-09-20T23:55:21.531 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:21 smithi137 ceph-mon[65321]: osdmap e151: 8 total, 7 up, 8 in 2024-09-20T23:55:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:21 smithi144 ceph-mon[51829]: pgmap v276: 97 pgs: 12 stale+active+clean, 85 active+clean; 339 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.8 KiB/s rd, 4.7 MiB/s wr, 474 op/s; 410 KiB/s, 0 objects/s recovering 2024-09-20T23:55:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:21 smithi144 ceph-mon[51829]: osdmap e151: 8 total, 7 up, 8 in 2024-09-20T23:55:21.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:21.351612492 +0000 UTC m=+0.378677503 container init a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, GIT_BRANCH=HEAD) 2024-09-20T23:55:21.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:21.3557454 +0000 UTC m=+0.382810410 container start a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:55:21.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:21.443183481 +0000 UTC m=+0.470248497 container attach a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-20T23:55:21.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:21 smithi144 podman[63130]: 2024-09-20 23:55:21.581101295 +0000 UTC m=+0.608166301 container died a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T23:55:22.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 260710 9.31 MB/sec execute 546 sec latency 50.621 ms 2024-09-20T23:55:22.510 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:22 smithi144 ceph-mon[51829]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-20T23:55:22.510 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:22 smithi144 ceph-mon[51829]: Health check failed: Degraded data redundancy: 120/1668 objects degraded (7.194%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:22.510 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 podman[63153]: 2024-09-20 23:55:22.248388404 +0000 UTC m=+0.657488432 container remove a3677cc76df6dfdba91eb27084372ef676603d9e8249f354c7fbe9112574eec7 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-20T23:55:22.510 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.7.service: Deactivated successfully. 2024-09-20T23:55:22.510 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 systemd[1]: Stopped Ceph osd.7 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:55:22.510 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.7.service: Consumed 6min 37.126s CPU time. 2024-09-20T23:55:22.546 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:22 smithi137 ceph-mon[65321]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-09-20T23:55:22.546 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:22 smithi137 ceph-mon[65321]: Health check failed: Degraded data redundancy: 120/1668 objects degraded (7.194%), 20 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:22.878 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 systemd[1]: Starting Ceph osd.7 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:55:22.879 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 podman[63252]: 2024-09-20 23:55:22.634088066 +0000 UTC m=+0.018455724 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:55:22.879 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 podman[63252]: 2024-09-20 23:55:22.75847542 +0000 UTC m=+0.142843111 container create 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=) 2024-09-20T23:55:23.164 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 podman[63252]: 2024-09-20 23:55:22.994631413 +0000 UTC m=+0.378999071 container init 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:55:23.164 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:22 smithi144 podman[63252]: 2024-09-20 23:55:22.999457499 +0000 UTC m=+0.383825149 container start 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-09-20T23:55:23.164 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 podman[63252]: 2024-09-20 23:55:23.09476554 +0000 UTC m=+0.479133205 container attach 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-09-20T23:55:23.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 261021 9.32 MB/sec execute 547 sec latency 65.036 ms 2024-09-20T23:55:23.445 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-mon[51829]: pgmap v278: 97 pgs: 4 active+undersized, 9 peering, 5 stale+active+clean, 20 active+undersized+degraded, 59 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 394 op/s; 120/1668 objects degraded (7.194%) 2024-09-20T23:55:23.446 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:55:23.446 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:55:23.446 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:55:23.446 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-20T23:55:23.565 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:23 smithi137 ceph-mon[65321]: pgmap v278: 97 pgs: 4 active+undersized, 9 peering, 5 stale+active+clean, 20 active+undersized+degraded, 59 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.0 KiB/s rd, 4.7 MiB/s wr, 394 op/s; 120/1668 objects degraded (7.194%) 2024-09-20T23:55:23.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T23:55:23.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T23:55:23.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T23:55:23.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-20T23:55:24.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 261446 9.30 MB/sec execute 548 sec latency 66.203 ms 2024-09-20T23:55:24.317 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T23:55:24.317 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-20T23:55:24.317 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T23:55:24.317 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-20T23:55:24.317 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate[63267]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:23 smithi144 bash[63252]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-20T23:55:24.318 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:24 smithi144 podman[63252]: 2024-09-20 23:55:24.009572251 +0000 UTC m=+1.393939912 container died 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:55:24.885 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:24 smithi144 podman[63393]: 2024-09-20 23:55:24.616749466 +0000 UTC m=+0.595495878 container remove 8506fb9abaa37062406103f52c49917c00c4fb88aa547415721aa804cee18f89 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-09-20T23:55:24.885 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:24 smithi144 podman[63450]: 2024-09-20 23:55:24.746747956 +0000 UTC m=+0.017393847 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:55:24.885 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:24 smithi144 podman[63450]: 2024-09-20 23:55:24.884568153 +0000 UTC m=+0.155214015 container create 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-09-20T23:55:25.140 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:25 smithi144 podman[63450]: 2024-09-20 23:55:25.112534009 +0000 UTC m=+0.383179886 container init 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-09-20T23:55:25.140 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:25 smithi144 podman[63450]: 2024-09-20 23:55:25.116207096 +0000 UTC m=+0.386852959 container start 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, io.buildah.version=1.37.2, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-20T23:55:25.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 261733 9.30 MB/sec execute 549 sec latency 60.144 ms 2024-09-20T23:55:25.414 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:25 smithi144 ceph-osd[63469]: -- 172.21.15.144:0/1791030637 <== mon.1 v2:172.21.15.144:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564c4269c1e0 con 0x564c41889c00 2024-09-20T23:55:25.415 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:25 smithi144 bash[63450]: 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 2024-09-20T23:55:25.415 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:25 smithi144 systemd[1]: Started Ceph osd.7 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:55:25.607 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:25 smithi137 ceph-mon[65321]: pgmap v279: 97 pgs: 4 active+undersized, 9 peering, 29 active+undersized+degraded, 55 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s wr, 355 op/s; 217/1650 objects degraded (13.152%) 2024-09-20T23:55:25.608 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:25.608 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:25.608 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:25 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:25 smithi144 ceph-mon[51829]: pgmap v279: 97 pgs: 4 active+undersized, 9 peering, 29 active+undersized+degraded, 55 active+clean; 341 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 4.2 MiB/s wr, 355 op/s; 217/1650 objects degraded (13.152%) 2024-09-20T23:55:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:25 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:26.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 261881 9.29 MB/sec execute 550 sec latency 60.701 ms 2024-09-20T23:55:26.628 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:26 smithi137 ceph-mon[65321]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-20T23:55:26.683 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:55:26.850 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:26 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:55:26.485+0000 7fa804f08640 -1 Falling back to public interface 2024-09-20T23:55:26.850 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:26 smithi144 ceph-mon[51829]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-09-20T23:55:27.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 262231 9.30 MB/sec execute 551 sec latency 50.745 ms 2024-09-20T23:55:27.543 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (8m) 2m ago 17m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (5m) 2m ago 17m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (5m) 33s ago 17m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (8m) 2m ago 17m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (11m) 2m ago 11m 40.8M - 17.2.0 e1d6a67b021e 27fbb1fba744 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (11m) 33s ago 11m 15.0M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (7m) 2m ago 18m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (7m) 33s ago 17m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (6m) 2m ago 19m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (6m) 33s ago 16m 44.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (11m) 2m ago 11m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (11m) 33s ago 11m 86.3M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:55:27.544 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (8m) 2m ago 17m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (8m) 33s ago 16m 16.5M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (4m) 2m ago 16m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (4m) 2m ago 15m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (3m) 2m ago 15m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (2m) 2m ago 14m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (110s) 33s ago 14m 307M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (74s) 33s ago 14m 254M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (38s) 33s ago 13m 12.1M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 starting - - - 2704M 2024-09-20T23:55:27.545 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (7m) 2m ago 16m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:55:27.637 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:27 smithi137 ceph-mon[65321]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.7 MiB/s wr, 399 op/s; 261/1584 objects degraded (16.477%) 2024-09-20T23:55:27.638 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:27 smithi137 ceph-mon[65321]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:27.838 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:27 smithi144 ceph-mon[51829]: pgmap v280: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 333 MiB data, 2.6 GiB used, 713 GiB / 715 GiB avail; 6.5 KiB/s rd, 3.7 MiB/s wr, 399 op/s; 261/1584 objects degraded (16.477%) 2024-09-20T23:55:27.838 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:27 smithi144 ceph-mon[51829]: from='client.34416 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.078 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 7 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:55:28.079 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:55:28.080 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2, 2024-09-20T23:55:28.080 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 11 2024-09-20T23:55:28.080 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:55:28.080 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:55:28.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 262518 9.30 MB/sec execute 552 sec latency 54.053 ms 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:55:28.528 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:55:28.529 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:55:28.529 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "13/23 daemons upgraded", 2024-09-20T23:55:28.529 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading osd daemons", 2024-09-20T23:55:28.529 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:55:28.529 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:55:28.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:28 smithi137 ceph-mon[65321]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:28 smithi137 ceph-mon[65321]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.657 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:28 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/212305008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:28.659 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:28 smithi144 ceph-mon[51829]: from='client.34420 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.659 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:28 smithi144 ceph-mon[51829]: from='client.34424 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:28.659 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:28 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/212305008' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:29.062 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_WARN 1 osds down; all OSDs are running squid or later but require_osd_release < squid; Degraded data redundancy: 265/1608 objects degraded (16.480%), 35 pgs degraded 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_DOWN: 1 osds down 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: osd.7 (root=default,host=smithi144) is down 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] OSD_UPGRADE_FINISHED: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: all OSDs are running squid or later but require_osd_release < squid 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout:[WRN] PG_DEGRADED: Degraded data redundancy: 265/1608 objects degraded (16.480%), 35 pgs degraded 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 1.0 is active+undersized+degraded, acting [0,6] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.0 is active+undersized+degraded, acting [1,0] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.3 is active+undersized+degraded, acting [5,2] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.4 is active+undersized+degraded, acting [1,0] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.5 is active+undersized+degraded, acting [0,4] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.7 is active+undersized+degraded, acting [6,2] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.8 is active+undersized+degraded, acting [0,1] 2024-09-20T23:55:29.063 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.9 is active+undersized+degraded, acting [1,3] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.a is active+undersized+degraded, acting [1,3] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.b is active+undersized+degraded, acting [4,5] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.e is active+undersized+degraded, acting [2,3] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.f is active+undersized+degraded, acting [4,0] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.12 is active+undersized+degraded, acting [5,3] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.18 is active+undersized+degraded, acting [5,4] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1a is active+undersized+degraded, acting [6,4] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1b is active+undersized+degraded, acting [3,6] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1d is active+undersized+degraded, acting [6,0] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 2.1f is active+undersized+degraded, acting [0,4] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.7 is active+undersized+degraded, acting [3,0] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.8 is active+undersized+degraded, acting [3,1] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.9 is active+undersized+degraded, acting [4,2] 2024-09-20T23:55:29.064 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.d is active+undersized+degraded, acting [5,6] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.e is active+undersized+degraded, acting [4,1] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.f is active+undersized+degraded, acting [4,0] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.11 is active+undersized+degraded, acting [4,6] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.12 is active+undersized+degraded, acting [0,3] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.13 is active+undersized+degraded, acting [4,2] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.14 is active+undersized+degraded, acting [4,6] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.15 is active+undersized+degraded, acting [3,4] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.16 is active+undersized+degraded, acting [5,1] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 3.1b is active+undersized+degraded, acting [0,4] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.0 is active+undersized+degraded, acting [3,0] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.5 is active+undersized+degraded, acting [6,4] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.a is active+undersized+degraded, acting [6,1] 2024-09-20T23:55:29.065 INFO:teuthology.orchestra.run.smithi137.stdout: pg 4.14 is active+undersized+degraded, acting [3,2] 2024-09-20T23:55:29.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 262804 9.30 MB/sec execute 553 sec latency 46.593 ms 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: pgmap v281: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 446 op/s; 265/1608 objects degraded (16.480%) 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 265/1608 objects degraded (16.480%), 35 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:29 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3791728347' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:55:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: pgmap v281: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.8 KiB/s rd, 4.1 MiB/s wr, 446 op/s; 265/1608 objects degraded (16.480%) 2024-09-20T23:55:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 265/1608 objects degraded (16.480%), 35 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: from='client.34432 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:55:29.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:29.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:29.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:29 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3791728347' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:55:30.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 263140 9.31 MB/sec execute 554 sec latency 49.849 ms 2024-09-20T23:55:30.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:30 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:30.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:30 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:30.945 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:30 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:30.945 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:30 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:31.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 263349 9.30 MB/sec execute 555 sec latency 52.513 ms 2024-09-20T23:55:31.846 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:31 smithi144 ceph-mon[51829]: pgmap v282: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 265/1608 objects degraded (16.480%) 2024-09-20T23:55:31.980 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:31 smithi137 ceph-mon[65321]: pgmap v282: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 338 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.7 MiB/s wr, 397 op/s; 265/1608 objects degraded (16.480%) 2024-09-20T23:55:32.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 263703 9.29 MB/sec execute 556 sec latency 39.602 ms 2024-09-20T23:55:33.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 263966 9.28 MB/sec execute 557 sec latency 44.150 ms 2024-09-20T23:55:34.026 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 508 op/s; 267/1635 objects degraded (16.330%) 2024-09-20T23:55:34.026 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-20T23:55:34.027 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T23:55:34.028 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-20T23:55:34.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: pgmap v283: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 339 MiB data, 2.7 GiB used, 713 GiB / 715 GiB avail; 7.6 KiB/s rd, 4.4 MiB/s wr, 508 op/s; 267/1635 objects degraded (16.330%) 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.0"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.1"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.2"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.3"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.4"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.5"}]': finished 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]: dispatch 2024-09-20T23:55:34.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.6"}]': finished 2024-09-20T23:55:34.196 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]: dispatch 2024-09-20T23:55:34.196 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd.7"}]': finished 2024-09-20T23:55:34.196 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd require-osd-release", "release": "squid"}]: dispatch 2024-09-20T23:55:34.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 264226 9.28 MB/sec execute 558 sec latency 55.039 ms 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all osd 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T23:55:35.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:34 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi137.qhqtcp"]}]: dispatch 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all osd 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: Upgrade: Setting require_osd_release to 19 squid 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: Health check cleared: OSD_UPGRADE_FINISHED (was: all OSDs are running squid or later but require_osd_release < squid) 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "osd require-osd-release", "release": "squid"}]': finished 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: osdmap e152: 8 total, 7 up, 8 in 2024-09-20T23:55:35.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:34 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi137.qhqtcp"]}]: dispatch 2024-09-20T23:55:35.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 264567 9.29 MB/sec execute 559 sec latency 60.303 ms 2024-09-20T23:55:35.783 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:35 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:55:35.490+0000 7fa804f08640 -1 osd.7 0 read_superblock omap replica is missing. 2024-09-20T23:55:35.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:35 smithi137 ceph-mon[65321]: pgmap v284: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 400 op/s; 274/1683 objects degraded (16.280%) 2024-09-20T23:55:35.786 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:35 smithi137 ceph-mon[65321]: Upgrade: It appears safe to stop mds.foofs.smithi137.qhqtcp 2024-09-20T23:55:35.786 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:35 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 274/1683 objects degraded (16.280%), 35 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:36.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:35 smithi144 ceph-mon[51829]: pgmap v284: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 340 MiB data, 2.7 GiB used, 712 GiB / 715 GiB avail; 5.2 KiB/s rd, 3.8 MiB/s wr, 400 op/s; 274/1683 objects degraded (16.280%) 2024-09-20T23:55:36.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:35 smithi144 ceph-mon[51829]: Upgrade: It appears safe to stop mds.foofs.smithi137.qhqtcp 2024-09-20T23:55:36.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:35 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 274/1683 objects degraded (16.280%), 35 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:36.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265015 9.28 MB/sec execute 560 sec latency 52.802 ms 2024-09-20T23:55:37.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265318 9.27 MB/sec execute 561 sec latency 38.994 ms 2024-09-20T23:55:37.443 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:36 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:55:36.988+0000 7fa804f08640 -1 osd.7 149 log_to_monitors true 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.0 MiB/s wr, 357 op/s; 258/1608 objects degraded (16.045%) 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: Upgrade: Updating mds.foofs.smithi137.qhqtcp 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: Deploying daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: from='osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:55:38.080 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:37 smithi137 ceph-mon[65321]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:55:38.193 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:55:37.876+0000 7fa7fccb3640 -1 osd.7 149 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-20T23:55:38.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: pgmap v286: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 334 MiB data, 2.8 GiB used, 712 GiB / 715 GiB avail; 7.1 KiB/s rd, 4.0 MiB/s wr, 357 op/s; 258/1608 objects degraded (16.045%) 2024-09-20T23:55:38.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: Upgrade: Updating mds.foofs.smithi137.qhqtcp 2024-09-20T23:55:38.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:38.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi137.qhqtcp", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:55:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:55:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: Deploying daemon mds.foofs.smithi137.qhqtcp on smithi137 2024-09-20T23:55:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: from='osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:55:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:37 smithi144 ceph-mon[51829]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-20T23:55:38.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265665 9.28 MB/sec execute 562 sec latency 40.059 ms 2024-09-20T23:55:38.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:38 smithi137 ceph-mon[65321]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T23:55:38.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:38 smithi137 ceph-mon[65321]: from='osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:38.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:38 smithi137 ceph-mon[65321]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T23:55:38.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:38 smithi137 ceph-mon[65321]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:39.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:38 smithi144 ceph-mon[51829]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-20T23:55:39.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:38 smithi144 ceph-mon[51829]: from='osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:39.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:38 smithi144 ceph-mon[51829]: osdmap e153: 8 total, 7 up, 8 in 2024-09-20T23:55:39.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:38 smithi144 ceph-mon[51829]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi144", "root=default"]}]: dispatch 2024-09-20T23:55:39.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.28 MB/sec execute 563 sec latency 462.232 ms 2024-09-20T23:55:40.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:39 smithi137 ceph-mon[65321]: pgmap v288: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 257/1587 objects degraded (16.194%) 2024-09-20T23:55:40.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:39 smithi137 ceph-mon[65321]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:55:40.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:39 smithi137 ceph-mon[65321]: osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377] boot 2024-09-20T23:55:40.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:39 smithi137 ceph-mon[65321]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T23:55:40.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:39 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:55:40.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:39 smithi144 ceph-mon[51829]: pgmap v288: 97 pgs: 7 active+undersized, 35 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 8.9 KiB/s rd, 5.7 MiB/s wr, 570 op/s; 257/1587 objects degraded (16.194%) 2024-09-20T23:55:40.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:39 smithi144 ceph-mon[51829]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-20T23:55:40.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:39 smithi144 ceph-mon[51829]: osd.7 [v2:172.21.15.144:6824/2571570377,v1:172.21.15.144:6825/2571570377] boot 2024-09-20T23:55:40.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:39 smithi144 ceph-mon[51829]: osdmap e154: 8 total, 8 up, 8 in 2024-09-20T23:55:40.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:39 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-20T23:55:40.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.26 MB/sec execute 564 sec latency 1462.259 ms 2024-09-20T23:55:40.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[65294]: 2024-09-20T23:55:40.184+0000 7f5af6f33640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:55:40.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:55:40.249+0000 7f803998c640 -1 bdev(0x557ec8faac00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 1 2024-09-20T23:55:40.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:55:40.253+0000 7f8037187640 -1 bdev(0x557ec8faac00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 16 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: Standby daemon mds.foofs.smithi144.ltxxzn assigned to filesystem foofs as rank 0 2024-09-20T23:55:41.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-20T23:55:41.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:55:41.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-09-20T23:55:41.179 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:40 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:replay} 2024-09-20T23:55:41.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: osdmap e155: 8 total, 8 up, 8 in 2024-09-20T23:55:41.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-20T23:55:41.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:55:41.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: osdmap e156: 8 total, 8 up, 8 in 2024-09-20T23:55:41.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: Standby daemon mds.foofs.smithi144.ltxxzn assigned to filesystem foofs as rank 0 2024-09-20T23:55:41.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-20T23:55:41.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:55:41.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-09-20T23:55:41.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:40 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:replay} 2024-09-20T23:55:41.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.25 MB/sec execute 565 sec latency 2462.288 ms 2024-09-20T23:55:42.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:41 smithi137 ceph-mon[65321]: pgmap v292: 97 pgs: 1 peering, 7 active+undersized, 34 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s wr, 477 op/s; 250/1587 objects degraded (15.753%) 2024-09-20T23:55:42.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:41 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 250/1587 objects degraded (15.753%), 34 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:42.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:41 smithi137 ceph-mon[65321]: osdmap e157: 8 total, 8 up, 8 in 2024-09-20T23:55:42.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:41 smithi144 ceph-mon[51829]: pgmap v292: 97 pgs: 1 peering, 7 active+undersized, 34 active+undersized+degraded, 55 active+clean; 335 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s wr, 477 op/s; 250/1587 objects degraded (15.753%) 2024-09-20T23:55:42.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:41 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 250/1587 objects degraded (15.753%), 34 pgs degraded (PG_DEGRADED) 2024-09-20T23:55:42.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:41 smithi144 ceph-mon[51829]: osdmap e157: 8 total, 8 up, 8 in 2024-09-20T23:55:42.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.23 MB/sec execute 566 sec latency 3462.341 ms 2024-09-20T23:55:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.21 MB/sec execute 567 sec latency 4462.373 ms 2024-09-20T23:55:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: osdmap e158: 8 total, 8 up, 8 in 2024-09-20T23:55:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:reconnect 2024-09-20T23:55:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:reconnect} 2024-09-20T23:55:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: reconnect by client.24351 172.21.15.144:0/3147319039 after 0.00399994 2024-09-20T23:55:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: reconnect by client.14542 172.21.15.137:0/3724698982 after 0.0449993 2024-09-20T23:55:43.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:43 smithi137 ceph-mon[65321]: pgmap v295: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 peering, 12 active+undersized+degraded, 77 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 402 op/s; 189/1608 objects degraded (11.754%); 2.4 MiB/s, 26 objects/s recovering 2024-09-20T23:55:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: osdmap e158: 8 total, 8 up, 8 in 2024-09-20T23:55:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:reconnect 2024-09-20T23:55:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:reconnect} 2024-09-20T23:55:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: reconnect by client.24351 172.21.15.144:0/3147319039 after 0.00399994 2024-09-20T23:55:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: reconnect by client.14542 172.21.15.137:0/3724698982 after 0.0449993 2024-09-20T23:55:43.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:43 smithi144 ceph-mon[51829]: pgmap v295: 97 pgs: 3 active+recovery_wait+undersized+degraded+remapped, 2 active+undersized+remapped, 3 peering, 12 active+undersized+degraded, 77 active+clean; 337 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 402 op/s; 189/1608 objects degraded (11.754%); 2.4 MiB/s, 26 objects/s recovering 2024-09-20T23:55:44.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265850 9.20 MB/sec execute 568 sec latency 5462.409 ms 2024-09-20T23:55:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:44 smithi137 ceph-mon[65321]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-20T23:55:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:44 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:rejoin 2024-09-20T23:55:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:44 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.137:6834/2310005421,v1:172.21.15.137:6835/2310005421] up:boot 2024-09-20T23:55:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:44 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:rejoin} 1 up:standby 2024-09-20T23:55:44.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:55:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:44 smithi144 ceph-mon[51829]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-20T23:55:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:44 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:rejoin 2024-09-20T23:55:44.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:44 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.137:6834/2310005421,v1:172.21.15.137:6835/2310005421] up:boot 2024-09-20T23:55:44.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:44 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:rejoin} 1 up:standby 2024-09-20T23:55:44.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:55:45.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 265865 9.18 MB/sec execute 569 sec latency 6398.715 ms 2024-09-20T23:55:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:45 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:clientreplay 2024-09-20T23:55:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:45 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:clientreplay} 1 up:standby 2024-09-20T23:55:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:45 smithi144 ceph-mon[51829]: daemon mds.foofs.smithi144.ltxxzn is now active in filesystem foofs as rank 0 2024-09-20T23:55:45.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:45 smithi144 ceph-mon[51829]: pgmap v296: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 peering, 93 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 3.2 MiB/s wr, 333 op/s; 23/1623 objects degraded (1.417%); 2.9 MiB/s, 65 objects/s recovering 2024-09-20T23:55:45.474 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:45 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:clientreplay 2024-09-20T23:55:45.474 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:45 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi144.ltxxzn=up:clientreplay} 1 up:standby 2024-09-20T23:55:45.474 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:45 smithi137 ceph-mon[65321]: daemon mds.foofs.smithi144.ltxxzn is now active in filesystem foofs as rank 0 2024-09-20T23:55:45.475 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:45 smithi137 ceph-mon[65321]: pgmap v296: 97 pgs: 1 active+recovery_wait+undersized+degraded+remapped, 1 active+undersized+remapped, 2 peering, 93 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 37 MiB/s rd, 3.2 MiB/s wr, 333 op/s; 23/1623 objects degraded (1.417%); 2.9 MiB/s, 65 objects/s recovering 2024-09-20T23:55:46.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:46 smithi137 ceph-mon[65321]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-20T23:55:46.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:46 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:active 2024-09-20T23:55:46.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:46 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi144.ltxxzn=up:active} 1 up:standby 2024-09-20T23:55:46.260 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:46 smithi137 ceph-mon[65321]: Health check update: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded (PG_DEGRADED) 2024-09-20T23:55:46.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266187 9.18 MB/sec execute 570 sec latency 67.523 ms 2024-09-20T23:55:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:46 smithi144 ceph-mon[51829]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-20T23:55:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:46 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.144:6832/4090375482,v1:172.21.15.144:6833/4090375482] up:active 2024-09-20T23:55:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:46 smithi144 ceph-mon[51829]: fsmap foofs:1 {0=foofs.smithi144.ltxxzn=up:active} 1 up:standby 2024-09-20T23:55:46.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:46 smithi144 ceph-mon[51829]: Health check update: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded (PG_DEGRADED) 2024-09-20T23:55:47.275 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:47 smithi137 ceph-mon[65321]: pgmap v297: 97 pgs: 2 peering, 95 active+clean; 338 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 2.3 MiB/s wr, 252 op/s; 2.3 MiB/s, 51 objects/s recovering 2024-09-20T23:55:47.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266480 9.19 MB/sec execute 571 sec latency 66.613 ms 2024-09-20T23:55:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:47 smithi144 ceph-mon[51829]: pgmap v297: 97 pgs: 2 peering, 95 active+clean; 338 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 2.3 MiB/s wr, 252 op/s; 2.3 MiB/s, 51 objects/s recovering 2024-09-20T23:55:48.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266657 9.18 MB/sec execute 572 sec latency 83.888 ms 2024-09-20T23:55:48.301 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:48 smithi137 ceph-mon[65321]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded) 2024-09-20T23:55:48.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:48 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:55:48.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:48.302 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:48 smithi144 ceph-mon[51829]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 23/1623 objects degraded (1.417%), 1 pg degraded) 2024-09-20T23:55:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:48 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:55:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:55:49.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266747 9.17 MB/sec execute 573 sec latency 103.441 ms 2024-09-20T23:55:49.346 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:49 smithi137 ceph-mon[65321]: pgmap v298: 97 pgs: 97 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 2.4 MiB/s wr, 259 op/s; 1.7 MiB/s, 39 objects/s recovering 2024-09-20T23:55:49.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:49 smithi144 ceph-mon[51829]: pgmap v298: 97 pgs: 97 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 26 MiB/s rd, 2.4 MiB/s wr, 259 op/s; 1.7 MiB/s, 39 objects/s recovering 2024-09-20T23:55:50.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266837 9.16 MB/sec execute 574 sec latency 142.558 ms 2024-09-20T23:55:51.288 INFO:teuthology.orchestra.run.smithi137.stdout: 5 266962 9.15 MB/sec execute 575 sec latency 149.448 ms 2024-09-20T23:55:51.396 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:51 smithi137 ceph-mon[65321]: pgmap v299: 97 pgs: 97 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.2 MiB/s wr, 95 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-09-20T23:55:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:51 smithi144 ceph-mon[51829]: pgmap v299: 97 pgs: 97 active+clean; 339 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 1.2 MiB/s wr, 95 op/s; 1.1 MiB/s, 29 objects/s recovering 2024-09-20T23:55:52.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 267160 9.13 MB/sec execute 576 sec latency 90.471 ms 2024-09-20T23:55:53.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 267342 9.12 MB/sec execute 577 sec latency 125.589 ms 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: pgmap v300: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 182 op/s; 593 KiB/s, 22 objects/s recovering 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: daemon mds.foofs.smithi137.qhqtcp restarted 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: osdmap e159: 8 total, 8 up, 8 in 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:53.558 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:53 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: pgmap v300: 97 pgs: 97 active+clean; 340 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 182 op/s; 593 KiB/s, 22 objects/s recovering 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: daemon mds.foofs.smithi137.qhqtcp restarted 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: osdmap e159: 8 total, 8 up, 8 in 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:53 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:55:54.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 267518 9.12 MB/sec execute 578 sec latency 88.766 ms 2024-09-20T23:55:54.574 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:54 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:boot 2024-09-20T23:55:54.575 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:54 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi144.ltxxzn=up:active} 1 up:standby 2024-09-20T23:55:54.575 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:55:54.575 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:54 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:54 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:boot 2024-09-20T23:55:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:54 smithi144 ceph-mon[51829]: fsmap foofs:1 {0=foofs.smithi144.ltxxzn=up:active} 1 up:standby 2024-09-20T23:55:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:55:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:54 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:55:55.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 267689 9.11 MB/sec execute 579 sec latency 82.837 ms 2024-09-20T23:55:55.588 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:55 smithi137 ceph-mon[65321]: pgmap v302: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 218 op/s; 110 KiB/s, 2 objects/s recovering 2024-09-20T23:55:55.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:55 smithi144 ceph-mon[51829]: pgmap v302: 97 pgs: 97 active+clean; 343 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.4 MiB/s rd, 2.0 MiB/s wr, 218 op/s; 110 KiB/s, 2 objects/s recovering 2024-09-20T23:55:56.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 267921 9.11 MB/sec execute 580 sec latency 94.729 ms 2024-09-20T23:55:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 268142 9.10 MB/sec execute 581 sec latency 77.763 ms 2024-09-20T23:55:57.671 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:57 smithi137 ceph-mon[65321]: pgmap v303: 97 pgs: 97 active+clean; 342 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-09-20T23:55:57.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:57 smithi144 ceph-mon[51829]: pgmap v303: 97 pgs: 97 active+clean; 342 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 29 KiB/s rd, 2.4 MiB/s wr, 239 op/s 2024-09-20T23:55:58.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 268339 9.09 MB/sec execute 582 sec latency 110.168 ms 2024-09-20T23:55:59.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 268676 9.08 MB/sec execute 583 sec latency 122.356 ms 2024-09-20T23:55:59.500 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:55:59 smithi137 ceph-mon[65321]: pgmap v304: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-09-20T23:55:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:55:59 smithi144 ceph-mon[51829]: pgmap v304: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 24 KiB/s rd, 2.7 MiB/s wr, 297 op/s 2024-09-20T23:55:59.718 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:56:00.290 INFO:teuthology.orchestra.run.smithi137.stdout: 5 268830 9.07 MB/sec execute 584 sec latency 70.922 ms 2024-09-20T23:56:00.545 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:00 smithi137 ceph-mon[65321]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:00.599 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:56:00.599 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (9m) 2m ago 18m 24.8M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:56:00.599 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (6m) 2m ago 18m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (6m) 31s ago 17m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (9m) 2m ago 18m 54.0M - dad864ee21e9 c491347d0b92 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 starting - - - - 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (12m) 31s ago 12m 15.1M - 17.2.0 e1d6a67b021e d460b8469291 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (8m) 2m ago 19m 504M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (7m) 31s ago 17m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (7m) 2m ago 19m 50.9M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (6m) 31s ago 17m 44.8M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (11m) 2m ago 11m 287M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (11m) 31s ago 11m 86.3M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (9m) 2m ago 17m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (8m) 31s ago 17m 16.5M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (5m) 2m ago 16m 330M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (4m) 2m ago 16m 258M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:56:00.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (3m) 2m ago 15m 205M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (3m) 2m ago 15m 11.3M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (2m) 31s ago 14m 346M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (107s) 31s ago 14m 304M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (71s) 31s ago 14m 242M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (35s) 31s ago 13m 11.5M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 386ad75da8a4 2024-09-20T23:56:00.601 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (7m) 2m ago 17m 64.5M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:56:00.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:00 smithi144 ceph-mon[51829]: from='client.34450 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:56:01.078 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 1 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 1, 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 13 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:56:01.079 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:56:01.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 268925 9.06 MB/sec execute 585 sec latency 99.601 ms 2024-09-20T23:56:01.591 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "crash" 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "15/23 daemons upgraded", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading mds daemons", 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:56:01.592 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: pgmap v305: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.9 MiB/s wr, 338 op/s 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/4087119052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:01.593 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:01 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: from='client.34454 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: pgmap v305: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 22 KiB/s rd, 2.9 MiB/s wr, 338 op/s 2024-09-20T23:56:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: from='client.34458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/4087119052' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:01.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:01 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:02.151 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:56:02.289 INFO:teuthology.orchestra.run.smithi137.stdout: 5 269065 9.05 MB/sec execute 586 sec latency 76.609 ms 2024-09-20T23:56:02.635 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:02 smithi137 ceph-mon[65321]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:02.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:02 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3130577644' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:56:02.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:02 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:02.636 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:02 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:02 smithi144 ceph-mon[51829]: from='client.34466 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:02 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3130577644' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:56:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:02 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:02.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:02 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:03.290 INFO:teuthology.orchestra.run.smithi137.stdout: 5 269344 9.06 MB/sec execute 587 sec latency 78.530 ms 2024-09-20T23:56:03.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:03 smithi137 ceph-mon[65321]: pgmap v306: 97 pgs: 97 active+clean; 331 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-09-20T23:56:03.417 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:03 smithi144 ceph-mon[51829]: pgmap v306: 97 pgs: 97 active+clean; 331 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 KiB/s rd, 2.5 MiB/s wr, 267 op/s 2024-09-20T23:56:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:04.299 INFO:teuthology.orchestra.run.smithi137.stdout: 5 269580 9.06 MB/sec execute 588 sec latency 90.987 ms 2024-09-20T23:56:05.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 269780 9.05 MB/sec execute 589 sec latency 94.327 ms 2024-09-20T23:56:05.668 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:05 smithi137 ceph-mon[65321]: pgmap v307: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.3 MiB/s wr, 286 op/s 2024-09-20T23:56:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:05 smithi144 ceph-mon[51829]: pgmap v307: 97 pgs: 97 active+clean; 334 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 2.3 MiB/s wr, 286 op/s 2024-09-20T23:56:06.299 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270075 9.06 MB/sec execute 590 sec latency 74.852 ms 2024-09-20T23:56:07.299 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270247 9.05 MB/sec execute 591 sec latency 69.521 ms 2024-09-20T23:56:07.489 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:07 smithi137 ceph-mon[65321]: pgmap v308: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-09-20T23:56:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:07 smithi144 ceph-mon[51829]: pgmap v308: 97 pgs: 97 active+clean; 335 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 2.5 MiB/s wr, 283 op/s 2024-09-20T23:56:08.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270314 9.04 MB/sec execute 592 sec latency 130.802 ms 2024-09-20T23:56:08.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: Detected new or changed devices on smithi137 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi144.ltxxzn"]}]: dispatch 2024-09-20T23:56:08.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:08 smithi144 ceph-mon[51829]: Upgrade: It appears safe to stop mds.foofs.smithi144.ltxxzn 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: Detected new or changed devices on smithi137 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.782 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:08.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds ok-to-stop", "ids": ["foofs.smithi144.ltxxzn"]}]: dispatch 2024-09-20T23:56:08.783 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:08 smithi137 ceph-mon[65321]: Upgrade: It appears safe to stop mds.foofs.smithi144.ltxxzn 2024-09-20T23:56:09.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270434 9.02 MB/sec execute 593 sec latency 115.643 ms 2024-09-20T23:56:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:09 smithi144 ceph-mon[51829]: pgmap v309: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-09-20T23:56:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:56:09.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:09 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:09.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:09 smithi137 ceph-mon[65321]: pgmap v309: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 7.7 KiB/s rd, 3.3 MiB/s wr, 336 op/s 2024-09-20T23:56:09.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:09.819 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi144.ltxxzn", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-09-20T23:56:09.820 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:09 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:10.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270576 9.02 MB/sec execute 594 sec latency 150.910 ms 2024-09-20T23:56:10.848 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:10 smithi137 ceph-mon[65321]: Upgrade: Updating mds.foofs.smithi144.ltxxzn 2024-09-20T23:56:10.848 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:10 smithi137 ceph-mon[65321]: Deploying daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:56:10.876 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:10 smithi144 ceph-mon[51829]: Upgrade: Updating mds.foofs.smithi144.ltxxzn 2024-09-20T23:56:10.876 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:10 smithi144 ceph-mon[51829]: Deploying daemon mds.foofs.smithi144.ltxxzn on smithi144 2024-09-20T23:56:11.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270804 9.00 MB/sec execute 595 sec latency 82.554 ms 2024-09-20T23:56:11.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[65294]: 2024-09-20T23:56:11.522+0000 7f5af6f33640 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:56:11.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: pgmap v310: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-09-20T23:56:11.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: Standby daemon mds.foofs.smithi137.qhqtcp assigned to filesystem foofs as rank 0 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-09-20T23:56:11.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:11 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:replay} 2024-09-20T23:56:11.943 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:56:11.541+0000 7f56824b2640 -1 bdev(0x5562143aac00 /var/lib/ceph/osd/ceph-6/block) aio_submit retries 1 2024-09-20T23:56:11.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:56:11.546+0000 7f8035984640 -1 bdev(0x557ec8faac00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 9 2024-09-20T23:56:11.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:56:11.546+0000 7f8034181640 -1 bdev(0x557ec8faac00 /var/lib/ceph/osd/ceph-5/block) aio_submit retries 11 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: pgmap v310: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.6 MiB/s wr, 242 op/s 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: osdmap e160: 8 total, 8 up, 8 in 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: Standby daemon mds.foofs.smithi137.qhqtcp assigned to filesystem foofs as rank 0 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: Health check failed: insufficient standby MDS daemons available (MDS_INSUFFICIENT_STANDBY) 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-09-20T23:56:11.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: fsmap foofs:0/1 1 up:standby, 1 failed 2024-09-20T23:56:11.945 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:11 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:replay} 2024-09-20T23:56:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270851 8.99 MB/sec execute 596 sec latency 795.620 ms 2024-09-20T23:56:13.308 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270851 8.98 MB/sec execute 597 sec latency 1802.907 ms 2024-09-20T23:56:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:13 smithi137 ceph-mon[65321]: pgmap v312: 97 pgs: 97 active+clean; 347 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-09-20T23:56:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:13 smithi137 ceph-mon[65321]: reconnect by client.24351 172.21.15.144:0/3147319039 after 0.000999983 2024-09-20T23:56:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:13 smithi137 ceph-mon[65321]: reconnect by client.14542 172.21.15.137:0/3724698982 after 0.000999983 2024-09-20T23:56:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:13 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:reconnect 2024-09-20T23:56:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:13 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:reconnect} 2024-09-20T23:56:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:13 smithi144 ceph-mon[51829]: pgmap v312: 97 pgs: 97 active+clean; 347 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 1.6 MiB/s rd, 3.3 MiB/s wr, 287 op/s 2024-09-20T23:56:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:13 smithi144 ceph-mon[51829]: reconnect by client.24351 172.21.15.144:0/3147319039 after 0.000999983 2024-09-20T23:56:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:13 smithi144 ceph-mon[51829]: reconnect by client.14542 172.21.15.137:0/3724698982 after 0.000999983 2024-09-20T23:56:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:13 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:reconnect 2024-09-20T23:56:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:13 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:reconnect} 2024-09-20T23:56:14.308 INFO:teuthology.orchestra.run.smithi137.stdout: 5 270851 8.96 MB/sec execute 598 sec latency 2802.936 ms 2024-09-20T23:56:14.891 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:14 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:rejoin 2024-09-20T23:56:14.892 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:14 smithi137 ceph-mon[65321]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:rejoin} 2024-09-20T23:56:14.892 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:14 smithi137 ceph-mon[65321]: daemon mds.foofs.smithi137.qhqtcp is now active in filesystem foofs as rank 0 2024-09-20T23:56:14.892 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:14 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:14 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:rejoin 2024-09-20T23:56:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:14 smithi144 ceph-mon[51829]: fsmap foofs:1/1 {0=foofs.smithi137.qhqtcp=up:rejoin} 2024-09-20T23:56:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:14 smithi144 ceph-mon[51829]: daemon mds.foofs.smithi137.qhqtcp is now active in filesystem foofs as rank 0 2024-09-20T23:56:14.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:14 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:15.308 INFO:teuthology.orchestra.run.smithi137.stdout: 5 271026 8.95 MB/sec execute 599 sec latency 3168.707 ms 2024-09-20T23:56:15.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:15 smithi137 ceph-mon[65321]: pgmap v313: 97 pgs: 97 active+clean; 344 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-09-20T23:56:15.655 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:15 smithi137 ceph-mon[65321]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-20T23:56:15.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:15 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:active 2024-09-20T23:56:15.656 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:15 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 2024-09-20T23:56:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:15 smithi144 ceph-mon[51829]: pgmap v313: 97 pgs: 97 active+clean; 344 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 9.2 MiB/s rd, 2.7 MiB/s wr, 200 op/s 2024-09-20T23:56:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:15 smithi144 ceph-mon[51829]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-09-20T23:56:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:15 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802] up:active 2024-09-20T23:56:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:15 smithi144 ceph-mon[51829]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 2024-09-20T23:56:16.309 INFO:teuthology.orchestra.run.smithi137.stdout: 5 cleanup 600 sec 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: 0 cleanup 600 sec 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Operation Count AvgLat MaxLat 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: ---------------------------------------- 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: NTCreateX 172188 4.908 3168.696 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Close 126448 0.131 11.679 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Rename 7308 11.170 6398.707 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Unlink 34818 7.320 2217.771 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Qpathinfo 156138 1.298 2317.969 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Qfileinfo 27228 0.006 2.042 2024-09-20T23:56:16.683 INFO:teuthology.orchestra.run.smithi137.stdout: Qfsinfo 28672 0.633 19.865 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: Sfileinfo 14063 13.626 1065.952 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: Find 60383 0.925 54.457 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: WriteX 85155 15.588 6381.286 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: ReadX 270329 0.010 9.429 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: LockX 562 0.529 19.964 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: UnlockX 562 0.424 18.499 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: Flush 12098 0.002 0.026 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:56:16.684 INFO:teuthology.orchestra.run.smithi137.stdout:Throughput 8.95487 MB/sec 5 clients 5 procs max_latency=6398.715 ms 2024-09-20T23:56:16.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'umount /mnt/foo' 2024-09-20T23:56:16.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done' 2024-09-20T23:56:16.847 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 5' 2024-09-20T23:56:16.891 INFO:teuthology.orchestra.run.smithi137.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-09-20T23:56:16.891 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:56:16.891 INFO:teuthology.orchestra.run.smithi137.stdout:Running for 5 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 1 secs 2024-09-20T23:56:16.891 INFO:teuthology.orchestra.run.smithi137.stdout:3 of 5 processes prepared for launch 0 sec 2024-09-20T23:56:16.992 INFO:teuthology.orchestra.run.smithi137.stdout:5 of 5 processes prepared for launch 0 sec 2024-09-20T23:56:16.992 INFO:teuthology.orchestra.run.smithi137.stdout:releasing clients 2024-09-20T23:56:17.773 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:17 smithi137 ceph-mon[65321]: pgmap v314: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 243 op/s 2024-09-20T23:56:17.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:17 smithi144 ceph-mon[51829]: pgmap v314: 97 pgs: 97 active+clean; 342 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 243 op/s 2024-09-20T23:56:18.992 INFO:teuthology.orchestra.run.smithi137.stdout: 5 371 21.07 MB/sec execute 1 sec latency 141.507 ms 2024-09-20T23:56:19.308 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:19.308 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:19.308 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:19 smithi137 ceph-mon[65321]: pgmap v315: 97 pgs: 97 active+clean; 348 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 235 op/s 2024-09-20T23:56:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:19 smithi144 ceph-mon[51829]: pgmap v315: 97 pgs: 97 active+clean; 348 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 235 op/s 2024-09-20T23:56:19.992 INFO:teuthology.orchestra.run.smithi137.stdout: 5 528 23.00 MB/sec execute 2 sec latency 109.884 ms 2024-09-20T23:56:20.992 INFO:teuthology.orchestra.run.smithi137.stdout: 5 838 20.85 MB/sec execute 3 sec latency 45.145 ms 2024-09-20T23:56:21.418 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:21 smithi137 ceph-mon[65321]: pgmap v316: 97 pgs: 97 active+clean; 348 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 241 op/s 2024-09-20T23:56:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:21 smithi144 ceph-mon[51829]: pgmap v316: 97 pgs: 97 active+clean; 348 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 23 MiB/s rd, 3.1 MiB/s wr, 241 op/s 2024-09-20T23:56:21.992 INFO:teuthology.orchestra.run.smithi137.stdout: 5 1182 17.49 MB/sec execute 4 sec latency 42.265 ms 2024-09-20T23:56:22.992 INFO:teuthology.orchestra.run.smithi137.stdout: 5 cleanup 5 sec 2024-09-20T23:56:23.160 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bd8600000034' failed - Device or resource busy 2024-09-20T23:56:23.160 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bc9c0000001c' failed - Device or resource busy 2024-09-20T23:56:23.161 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bca500000044' failed - Device or resource busy 2024-09-20T23:56:23.161 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bc9f00000025' failed - Device or resource busy 2024-09-20T23:56:23.161 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bc9900000042' failed - Device or resource busy 2024-09-20T23:56:23.161 INFO:teuthology.orchestra.run.smithi137.stdout:[1528] unlink './clients/client4/~dmtmp/PARADOX/.nfs000001000000bd8b0000002b' failed - Device or resource busy 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: 0 cleanup 5 sec 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Operation Count AvgLat MaxLat 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: ---------------------------------------- 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: NTCreateX 1041 6.296 141.501 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Close 877 7.414 115.099 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Rename 50 4.992 40.925 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Unlink 102 9.188 48.663 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Qpathinfo 664 1.438 35.024 2024-09-20T23:56:23.244 INFO:teuthology.orchestra.run.smithi137.stdout: Qfileinfo 383 14.559 113.364 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: Qfsinfo 165 0.377 0.763 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: Sfileinfo 170 13.390 29.229 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: Find 283 0.718 2.258 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: WriteX 1677 0.025 0.082 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: ReadX 1228 0.014 0.620 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: Flush 129 12.789 26.950 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:56:23.245 INFO:teuthology.orchestra.run.smithi137.stdout:Throughput 17.4881 MB/sec 5 clients 5 procs max_latency=141.507 ms 2024-09-20T23:56:23.246 DEBUG:teuthology.parallel:result is None 2024-09-20T23:56:23.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:23 smithi137 ceph-mon[65321]: pgmap v317: 97 pgs: 97 active+clean; 340 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 9.8 MiB/s wr, 354 op/s 2024-09-20T23:56:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:23 smithi144 ceph-mon[51829]: pgmap v317: 97 pgs: 97 active+clean; 340 MiB data, 2.4 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 9.8 MiB/s wr, 354 op/s 2024-09-20T23:56:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-20T23:56:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: Cluster is now healthy 2024-09-20T23:56:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: mds.? [v2:172.21.15.144:6832/4241523892,v1:172.21.15.144:6833/4241523892] up:boot 2024-09-20T23:56:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:56:24.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:24.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:24.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:24.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:24 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: Health check cleared: MDS_INSUFFICIENT_STANDBY (was: insufficient standby MDS daemons available) 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: Cluster is now healthy 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: mds.? [v2:172.21.15.144:6832/4241523892,v1:172.21.15.144:6833/4241523892] up:boot 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: fsmap foofs:1 {0=foofs.smithi137.qhqtcp=up:active} 1 up:standby 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:24.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:24 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:25.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:25 smithi137 ceph-mon[65321]: pgmap v318: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 10 MiB/s wr, 353 op/s 2024-09-20T23:56:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:25 smithi144 ceph-mon[51829]: pgmap v318: 97 pgs: 97 active+clean; 324 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 18 MiB/s rd, 10 MiB/s wr, 353 op/s 2024-09-20T23:56:27.546 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:27 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.546 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:27 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.546 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:27 smithi144 ceph-mon[51829]: pgmap v319: 97 pgs: 97 active+clean; 289 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 11 MiB/s wr, 376 op/s 2024-09-20T23:56:27.546 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:27 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.546 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:27 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:27 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:27 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:27 smithi137 ceph-mon[65321]: pgmap v319: 97 pgs: 97 active+clean; 289 MiB data, 2.5 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 11 MiB/s wr, 376 op/s 2024-09-20T23:56:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:27 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:27 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:29 smithi137 ceph-mon[65321]: pgmap v320: 97 pgs: 97 active+clean; 237 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 11 MiB/s wr, 377 op/s 2024-09-20T23:56:29.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:29 smithi144 ceph-mon[51829]: pgmap v320: 97 pgs: 97 active+clean; 237 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 11 MiB/s wr, 377 op/s 2024-09-20T23:56:31.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Detected new or changed devices on smithi144 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: pgmap v321: 97 pgs: 97 active+clean; 237 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 9.1 MiB/s wr, 307 op/s 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all mds 2024-09-20T23:56:31.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi137.qhqtcp"}]': finished 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi144.ltxxzn"}]': finished 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all rgw 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all rbd-mirror 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all ceph-exporter 2024-09-20T23:56:31.430 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.431 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.431 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all iscsi 2024-09-20T23:56:31.431 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:31 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Detected new or changed devices on smithi144 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: pgmap v321: 97 pgs: 97 active+clean; 237 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 28 KiB/s rd, 9.1 MiB/s wr, 307 op/s 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:31.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all mds 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi137.qhqtcp"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi137.qhqtcp"}]': finished 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi144.ltxxzn"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds.foofs.smithi144.ltxxzn"}]': finished 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all rgw 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all rbd-mirror 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all ceph-exporter 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all iscsi 2024-09-20T23:56:31.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:31 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:32.533 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Upgrade: Updating nfs.foo.0.0.smithi137.gcqarv (1/2) 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:33.044 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Rados config object exists: conf-nfs.foo 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv-rgw 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Bind address in nfs.foo.0.0.smithi137.gcqarv's ganesha conf is defaulting to empty 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:33.045 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:32 smithi137 ceph-mon[65321]: Deploying daemon nfs.foo.0.0.smithi137.gcqarv on smithi137 2024-09-20T23:56:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Upgrade: Updating nfs.foo.0.0.smithi137.gcqarv (1/2) 2024-09-20T23:56:33.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Rados config object exists: conf-nfs.foo 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Creating key for client.nfs.foo.0.0.smithi137.gcqarv-rgw 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi137.gcqarv-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Bind address in nfs.foo.0.0.smithi137.gcqarv's ganesha conf is defaulting to empty 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:33.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:32 smithi144 ceph-mon[51829]: Deploying daemon nfs.foo.0.0.smithi137.gcqarv on smithi137 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (9m) 31s ago 18m 25.7M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (6m) 31s ago 18m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (6m) 7s ago 18m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (9m) 31s ago 18m 54.8M - dad864ee21e9 c491347d0b92 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (41s) 31s ago 12m 12.9M - 19.1.1-367-g9e2ac70b cc9251b744a3 a7a5a9b87ed4 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (10s) 7s ago 12m 12.6M - 19.1.1-367-g9e2ac70b cc9251b744a3 a48eb0686b29 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (9m) 31s ago 20m 508M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:56:33.255 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (8m) 7s ago 18m 442M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (7m) 31s ago 20m 59.0M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (7m) 7s ago 18m 50.7M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (12m) 31s ago 12m 284M - 4.0 e1d6a67b021e b2c9d8429493 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (12m) 7s ago 12m 86.5M - 4.0 e1d6a67b021e c71eec772500 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (9m) 31s ago 18m 16.7M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (9m) 7s ago 17m 17.3M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (5m) 31s ago 17m 386M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (5m) 31s ago 16m 403M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (4m) 31s ago 16m 351M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (3m) 31s ago 15m 372M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (2m) 7s ago 15m 374M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (2m) 7s ago 15m 351M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (104s) 7s ago 14m 299M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:56:33.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (68s) 7s ago 14m 331M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 386ad75da8a4 2024-09-20T23:56:33.257 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (8m) 31s ago 17m 66.8M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:56:33.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 14 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:56:33.683 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:56:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: pgmap v322: 97 pgs: 97 active+clean; 229 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.2 MiB/s wr, 307 op/s 2024-09-20T23:56:34.006 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:34.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:34.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:34.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:34.007 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:33 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2652598540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:34.066 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:56:34.066 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:56:34.066 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:56:34.066 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "crash", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "mds", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "osd", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "16/23 daemons upgraded", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading nfs daemons", 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:56:34.067 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:56:34.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: pgmap v322: 97 pgs: 97 active+clean; 229 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 9.2 MiB/s wr, 307 op/s 2024-09-20T23:56:34.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: from='client.34486 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:34.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: from='client.34490 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:34.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:34.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:33 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2652598540' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:34.481 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:56:35.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:34 smithi137 ceph-mon[65321]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:35.040 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:34 smithi137 ceph-mon[65321]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:35.041 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:34 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/775952929' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:56:35.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:34 smithi144 ceph-mon[51829]: from='client.34494 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:35.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:34 smithi144 ceph-mon[51829]: from='client.34502 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:56:35.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:34 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/775952929' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:56:36.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:35 smithi137 ceph-mon[65321]: pgmap v323: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 166 op/s 2024-09-20T23:56:36.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:35 smithi137 ceph-mon[65321]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.137:0/3724698982"}]: dispatch 2024-09-20T23:56:36.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:35 smithi144 ceph-mon[51829]: pgmap v323: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 4.2 KiB/s rd, 2.7 MiB/s wr, 166 op/s 2024-09-20T23:56:36.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:35 smithi144 ceph-mon[51829]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.137:0/3724698982"}]: dispatch 2024-09-20T23:56:37.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:36 smithi144 ceph-mon[51829]: Evicting (and blocklisting) client session 14542 (172.21.15.137:0/3724698982) 2024-09-20T23:56:37.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:36 smithi144 ceph-mon[51829]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.137:0/3724698982"}]': finished 2024-09-20T23:56:37.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:36 smithi144 ceph-mon[51829]: osdmap e161: 8 total, 8 up, 8 in 2024-09-20T23:56:37.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:36 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:37.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:36 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:36 smithi137 ceph-mon[65321]: Evicting (and blocklisting) client session 14542 (172.21.15.137:0/3724698982) 2024-09-20T23:56:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:36 smithi137 ceph-mon[65321]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.137:0/3724698982"}]': finished 2024-09-20T23:56:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:36 smithi137 ceph-mon[65321]: osdmap e161: 8 total, 8 up, 8 in 2024-09-20T23:56:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:36 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:37.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:36 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: pgmap v325: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 389 KiB/s wr, 73 op/s 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:56:38.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:38 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: pgmap v325: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 614 B/s rd, 389 KiB/s wr, 73 op/s 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:38.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-09-20T23:56:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-09-20T23:56:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi144.kqljhb-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-20T23:56:38.429 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:38 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Upgrade: Updating nfs.foo.1.0.smithi144.kqljhb (2/2) 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Rados config object exists: conf-nfs.foo 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb-rgw 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Bind address in nfs.foo.1.0.smithi144.kqljhb's ganesha conf is defaulting to empty 2024-09-20T23:56:39.271 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:39 smithi144 ceph-mon[51829]: Deploying daemon nfs.foo.1.0.smithi144.kqljhb on smithi144 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Upgrade: Updating nfs.foo.1.0.smithi144.kqljhb (2/2) 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Rados config object exists: conf-nfs.foo 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Creating key for client.nfs.foo.1.0.smithi144.kqljhb-rgw 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Bind address in nfs.foo.1.0.smithi144.kqljhb's ganesha conf is defaulting to empty 2024-09-20T23:56:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:39 smithi137 ceph-mon[65321]: Deploying daemon nfs.foo.1.0.smithi144.kqljhb on smithi144 2024-09-20T23:56:40.342 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:40 smithi144 ceph-mon[51829]: pgmap v326: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-09-20T23:56:40.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:40 smithi137 ceph-mon[65321]: pgmap v326: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-09-20T23:56:42.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:42 smithi137 ceph-mon[65321]: pgmap v327: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-09-20T23:56:42.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:42 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:42.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:42 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:42.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:42 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:42 smithi144 ceph-mon[51829]: pgmap v327: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 2.8 KiB/s rd, 43 KiB/s wr, 9 op/s 2024-09-20T23:56:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:42 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:42 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:42.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:42 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:43.397 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:43 smithi144 ceph-mon[51829]: pgmap v328: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-09-20T23:56:43.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:43 smithi137 ceph-mon[65321]: pgmap v328: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 3.9 KiB/s wr, 8 op/s 2024-09-20T23:56:44.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:44.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:44.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: Evicting (and blocklisting) client session 24351 (172.21.15.144:0/3147319039) 2024-09-20T23:56:44.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.144:0/3147319039"}]: dispatch 2024-09-20T23:56:44.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:44.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:44 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:45.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:45.186 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:45.187 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: Evicting (and blocklisting) client session 24351 (172.21.15.144:0/3147319039) 2024-09-20T23:56:45.187 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd=[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.144:0/3147319039"}]: dispatch 2024-09-20T23:56:45.187 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:45.187 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:44 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:46.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:45 smithi137 ceph-mon[65321]: pgmap v329: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-09-20T23:56:46.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:45 smithi137 ceph-mon[65321]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.144:0/3147319039"}]': finished 2024-09-20T23:56:46.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:45 smithi137 ceph-mon[65321]: osdmap e162: 8 total, 8 up, 8 in 2024-09-20T23:56:46.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:45 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:46.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:45 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:46.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:45 smithi144 ceph-mon[51829]: pgmap v329: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.0 KiB/s rd, 4.0 KiB/s wr, 7 op/s 2024-09-20T23:56:46.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:45 smithi144 ceph-mon[51829]: from='mds.? [v2:172.21.15.137:6834/2996590802,v1:172.21.15.137:6835/2996590802]' entity='mds.foofs.smithi137.qhqtcp' cmd='[{"prefix":"osd blocklist", "blocklistop":"add","addr":"172.21.15.144:0/3147319039"}]': finished 2024-09-20T23:56:46.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:45 smithi144 ceph-mon[51829]: osdmap e162: 8 total, 8 up, 8 in 2024-09-20T23:56:46.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:45 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:46.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:45 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:47 smithi137 ceph-mon[65321]: pgmap v331: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-09-20T23:56:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:47 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:47 smithi144 ceph-mon[51829]: pgmap v331: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 5.1 KiB/s rd, 4.1 KiB/s wr, 8 op/s 2024-09-20T23:56:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:47 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:49.048 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:49.049 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:49.049 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:49.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: pgmap v332: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:49 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: pgmap v332: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:49 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all nfs 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi137.gcqarv"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi137.gcqarv"}]': finished 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi144.kqljhb"}]: dispatch 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi144.kqljhb"}]': finished 2024-09-20T23:56:50.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: Upgrade: Setting container_image for all nvmeof 2024-09-20T23:56:50.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:50.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:50.930 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:50 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all nfs 2024-09-20T23:56:51.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi137.gcqarv"}]: dispatch 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.0.0.smithi137.gcqarv"}]': finished 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi144.kqljhb"}]: dispatch 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs.foo.1.0.smithi144.kqljhb"}]': finished 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: Upgrade: Setting container_image for all nvmeof 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:51.195 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:50 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:56:52.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:51 smithi137 ceph-mon[65321]: pgmap v333: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-20T23:56:52.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:51 smithi144 ceph-mon[51829]: pgmap v333: 97 pgs: 97 active+clean; 226 MiB data, 2.2 GiB used, 713 GiB / 715 GiB avail; 3.0 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-09-20T23:56:53.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:52 smithi137 ceph-mon[65321]: Upgrade: Updating grafana.smithi137 2024-09-20T23:56:53.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:52 smithi137 ceph-mon[65321]: Deploying daemon grafana.smithi137 on smithi137 2024-09-20T23:56:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:52 smithi144 ceph-mon[51829]: Upgrade: Updating grafana.smithi137 2024-09-20T23:56:53.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:52 smithi144 ceph-mon[51829]: Deploying daemon grafana.smithi137 on smithi137 2024-09-20T23:56:54.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:53 smithi137 ceph-mon[65321]: pgmap v334: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 614 B/s wr, 1 op/s 2024-09-20T23:56:54.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:53 smithi144 ceph-mon[51829]: pgmap v334: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 614 B/s wr, 1 op/s 2024-09-20T23:56:56.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:55 smithi137 ceph-mon[65321]: pgmap v335: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 0 op/s 2024-09-20T23:56:56.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:55 smithi144 ceph-mon[51829]: pgmap v335: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 204 B/s rd, 511 B/s wr, 0 op/s 2024-09-20T23:56:58.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:56:57 smithi137 ceph-mon[65321]: pgmap v336: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 179 B/s rd, 448 B/s wr, 0 op/s 2024-09-20T23:56:58.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:56:57 smithi144 ceph-mon[51829]: pgmap v336: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 179 B/s rd, 448 B/s wr, 0 op/s 2024-09-20T23:57:00.383 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:00 smithi137 ceph-mon[65321]: pgmap v337: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-09-20T23:57:00.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:00 smithi144 ceph-mon[51829]: pgmap v337: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 341 B/s wr, 0 op/s 2024-09-20T23:57:02.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:02 smithi137 ceph-mon[65321]: pgmap v338: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:02.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:02 smithi144 ceph-mon[51829]: pgmap v338: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:03.935 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:03 smithi137 ceph-mon[65321]: pgmap v339: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:03.935 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:03.936 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:03.936 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:03.936 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:03 smithi144 ceph-mon[51829]: pgmap v339: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:04.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:04.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:04.860 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:57:05.545 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (10m) 19s ago 19m 25.1M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (7m) 19s ago 19m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (7m) 21s ago 18m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 starting - - - - 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (73s) 19s ago 13m 37.1M - 19.1.1-367-g9e2ac70b cc9251b744a3 a7a5a9b87ed4 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (42s) 21s ago 13m 12.7M - 19.1.1-367-g9e2ac70b cc9251b744a3 a48eb0686b29 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (9m) 19s ago 20m 510M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (8m) 21s ago 18m 443M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (8m) 19s ago 20m 61.4M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (8m) 21s ago 18m 47.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (30s) 19s ago 13m 62.0M - 5.5 cc9251b744a3 dde28899e48b 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (24s) 21s ago 12m 19.5M - 5.5 cc9251b744a3 dd3eeb69acf3 2024-09-20T23:57:05.546 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (10m) 19s ago 19m 16.9M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (9m) 21s ago 18m 17.0M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (6m) 19s ago 17m 408M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (5m) 19s ago 17m 412M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (4m) 19s ago 16m 378M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (4m) 19s ago 16m 398M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (3m) 21s ago 16m 375M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (2m) 21s ago 15m 351M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (2m) 21s ago 15m 297M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (100s) 21s ago 14m 331M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 386ad75da8a4 2024-09-20T23:57:05.547 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (8m) 19s ago 18m 66.8M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:57:05.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:05 smithi137 ceph-mon[65321]: pgmap v340: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:05.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:05 smithi137 ceph-mon[65321]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:05.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 14 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:57:05.972 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:57:06.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:05 smithi144 ceph-mon[51829]: pgmap v340: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:06.194 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:05 smithi144 ceph-mon[51829]: from='client.34534 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:06.313 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:57:06.313 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-20T23:57:06.313 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": true, 2024-09-20T23:57:06.313 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "Upgrading all daemon types on all hosts", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [ 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "crash", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mds", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "osd", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "nfs", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mon", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr" 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": "18/23 daemons upgraded", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "Currently upgrading grafana daemons", 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:57:06.314 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:57:06.748 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:57:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:06 smithi137 ceph-mon[65321]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:06 smithi137 ceph-mon[65321]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:07.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:06 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3008756407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:06 smithi144 ceph-mon[51829]: from='client.34538 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:06 smithi144 ceph-mon[51829]: from='client.34542 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:07.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:06 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3008756407' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:07 smithi137 ceph-mon[65321]: pgmap v341: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:07 smithi137 ceph-mon[65321]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:08.178 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:07 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/854368710' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:57:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:07 smithi144 ceph-mon[51829]: pgmap v341: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:07 smithi144 ceph-mon[51829]: from='client.34550 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:08.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:07 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/854368710' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:57:09.984 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:09 smithi137 ceph-mon[65321]: pgmap v342: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:10.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:09 smithi144 ceph-mon[51829]: pgmap v342: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:11.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.443 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:11 smithi144 ceph-mon[51829]: pgmap v343: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:11.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.444 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:11 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.521 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:11 smithi137 ceph-mon[65321]: pgmap v343: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:11.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:11.522 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:11 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:13.599 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:13 smithi137 ceph-mon[65321]: pgmap v344: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:13.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:13 smithi144 ceph-mon[51829]: pgmap v344: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: pgmap v345: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:57:15.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.680 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: Upgrade: Finalizing container_image settings 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-20T23:57:15.681 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-20T23:57:15.682 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: Upgrade: Complete! 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.683 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.684 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: pgmap v345: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.694 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.695 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: Upgrade: Finalizing container_image settings 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mgr"}]': finished 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mon"}]': finished 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.crash"}]': finished 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "osd"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "osd"}]': finished 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mds"}]: dispatch 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "mds"}]': finished 2024-09-20T23:57:15.696 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rgw"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.rbd-mirror"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.ceph-exporter"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.iscsi"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nfs"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix": "config rm", "name": "container_image", "who": "client.nvmeof"}]': finished 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.697 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config rm", "name": "container_image", "who": "mon"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: Upgrade: Complete! 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd='[{"prefix":"config-key del","key":"mgr/cephadm/upgrade_state"}]': finished 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:57:15.698 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:57:15.699 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:15.699 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:17.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:17 smithi137 ceph-mon[65321]: pgmap v346: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:17.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:17 smithi144 ceph-mon[51829]: pgmap v346: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:18.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:19.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:19 smithi137 ceph-mon[65321]: pgmap v347: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:19.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:19 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:19 smithi144 ceph-mon[51829]: pgmap v347: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:19 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:57:21.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:21 smithi137 ceph-mon[65321]: pgmap v348: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:21 smithi144 ceph-mon[51829]: pgmap v348: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:23.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:23 smithi137 ceph-mon[65321]: pgmap v349: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:23 smithi144 ceph-mon[51829]: pgmap v349: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:25.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:25 smithi137 ceph-mon[65321]: pgmap v350: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:25 smithi144 ceph-mon[51829]: pgmap v350: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:27 smithi137 ceph-mon[65321]: pgmap v351: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:27 smithi144 ceph-mon[51829]: pgmap v351: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:29 smithi137 ceph-mon[65321]: pgmap v352: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:29.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:29 smithi144 ceph-mon[51829]: pgmap v352: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:31.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:31 smithi137 ceph-mon[65321]: pgmap v353: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:31 smithi144 ceph-mon[51829]: pgmap v353: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:33.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:33 smithi137 ceph-mon[65321]: pgmap v354: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:33.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:33 smithi144 ceph-mon[51829]: pgmap v354: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:35 smithi137 ceph-mon[65321]: pgmap v355: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:35 smithi144 ceph-mon[51829]: pgmap v355: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:37 smithi137 ceph-mon[65321]: pgmap v356: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:37 smithi137 ceph-mon[65321]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:37 smithi144 ceph-mon[51829]: pgmap v356: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:37 smithi144 ceph-mon[51829]: from='client.34558 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:37.834 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch ps' 2024-09-20T23:57:39.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:39 smithi137 ceph-mon[65321]: pgmap v357: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:39.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:39 smithi144 ceph-mon[51829]: pgmap v357: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:39.925 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (11m) 29s ago 20m 25.1M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (7m) 29s ago 19m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (7m) 56s ago 19m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (37s) 29s ago 19m 41.9M - 9.4.12 2bacad6d85d8 fef4813f60ef 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (107s) 29s ago 13m 37.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 a7a5a9b87ed4 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (77s) 56s ago 13m 12.7M - 19.1.1-367-g9e2ac70b cc9251b744a3 a48eb0686b29 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (10m) 29s ago 21m 517M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (9m) 56s ago 19m 443M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (8m) 29s ago 21m 62.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (8m) 56s ago 19m 47.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:57:39.926 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (64s) 29s ago 13m 62.4M - 5.5 cc9251b744a3 dde28899e48b 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (59s) 56s ago 13m 19.5M - 5.5 cc9251b744a3 dd3eeb69acf3 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (10m) 29s ago 19m 16.6M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (10m) 56s ago 19m 17.0M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (7m) 29s ago 18m 409M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (6m) 29s ago 17m 413M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (5m) 29s ago 17m 379M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (4m) 29s ago 17m 397M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (4m) 56s ago 16m 375M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (3m) 56s ago 16m 351M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (2m) 56s ago 15m 297M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (2m) 56s ago 15m 331M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 386ad75da8a4 2024-09-20T23:57:39.927 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (9m) 29s ago 18m 66.8M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:57:40.616 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:40 smithi137 ceph-mon[65321]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:40.639 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions' 2024-09-20T23:57:40.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:40 smithi144 ceph-mon[51829]: from='client.34562 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:57:41.598 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:41 smithi137 ceph-mon[65321]: pgmap v358: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:41 smithi144 ceph-mon[51829]: pgmap v358: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:57:42.703 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 14 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:57:42.704 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:57:43.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'echo "wait for servicemap items w/ changing names to refresh"' 2024-09-20T23:57:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:43 smithi137 ceph-mon[65321]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:43 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2532266631' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:43 smithi144 ceph-mon[51829]: pgmap v359: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:43 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2532266631' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:57:45.249 INFO:teuthology.orchestra.run.smithi137.stdout:wait for servicemap items w/ changing names to refresh 2024-09-20T23:57:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:45 smithi137 ceph-mon[65321]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:45 smithi144 ceph-mon[51829]: pgmap v360: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:45.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'sleep 60' 2024-09-20T23:57:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:47 smithi137 ceph-mon[65321]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:47 smithi144 ceph-mon[51829]: pgmap v361: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:48.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:48.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:57:49.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:49 smithi137 ceph-mon[65321]: pgmap v362: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:49.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:49 smithi144 ceph-mon[51829]: pgmap v362: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:51.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:51 smithi137 ceph-mon[65321]: pgmap v363: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:51.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:51 smithi144 ceph-mon[51829]: pgmap v363: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:53.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:53 smithi137 ceph-mon[65321]: pgmap v364: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:53.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:53 smithi144 ceph-mon[51829]: pgmap v364: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:55.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:55 smithi137 ceph-mon[65321]: pgmap v365: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:55.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:55 smithi144 ceph-mon[51829]: pgmap v365: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:57.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:57 smithi137 ceph-mon[65321]: pgmap v366: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:57.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:57 smithi144 ceph-mon[51829]: pgmap v366: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:57:59.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:57:59 smithi137 ceph-mon[65321]: pgmap v367: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:57:59.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:57:59 smithi144 ceph-mon[51829]: pgmap v367: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:01.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:01 smithi137 ceph-mon[65321]: pgmap v368: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:01.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:01 smithi144 ceph-mon[51829]: pgmap v368: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:03.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:03 smithi137 ceph-mon[65321]: pgmap v369: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:03.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:03 smithi144 ceph-mon[51829]: pgmap v369: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:03.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:05.745 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:05 smithi144 ceph-mon[51829]: pgmap v370: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:05.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:05 smithi137 ceph-mon[65321]: pgmap v370: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:07 smithi137 ceph-mon[65321]: pgmap v371: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:07.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:07 smithi144 ceph-mon[51829]: pgmap v371: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:09.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:09 smithi137 ceph-mon[65321]: pgmap v372: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:09 smithi144 ceph-mon[51829]: pgmap v372: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:11.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:11 smithi137 ceph-mon[65321]: pgmap v373: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:11 smithi144 ceph-mon[51829]: pgmap v373: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:13.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:13 smithi137 ceph-mon[65321]: pgmap v374: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:13 smithi144 ceph-mon[51829]: pgmap v374: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:15.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: pgmap v375: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:15.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:58:15.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:58:15.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:58:15.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.929 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:15 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: pgmap v375: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:58:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:58:15.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:58:15.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:15.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:15 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:58:17.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:17 smithi137 ceph-mon[65321]: pgmap v376: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:17.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:17 smithi144 ceph-mon[51829]: pgmap v376: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:18.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:18 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:18.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:18 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:19.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:19 smithi137 ceph-mon[65321]: pgmap v377: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:19.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:19 smithi144 ceph-mon[51829]: pgmap v377: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:21.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:21 smithi137 ceph-mon[65321]: pgmap v378: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:21.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:21 smithi144 ceph-mon[51829]: pgmap v378: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:23.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:23 smithi137 ceph-mon[65321]: pgmap v379: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:23.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:23 smithi144 ceph-mon[51829]: pgmap v379: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:25.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:25 smithi137 ceph-mon[65321]: pgmap v380: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:25.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:25 smithi144 ceph-mon[51829]: pgmap v380: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:27.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:27 smithi137 ceph-mon[65321]: pgmap v381: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:27.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:27 smithi144 ceph-mon[51829]: pgmap v381: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:29.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:29 smithi137 ceph-mon[65321]: pgmap v382: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:29.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:29 smithi144 ceph-mon[51829]: pgmap v382: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:31.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:31 smithi137 ceph-mon[65321]: pgmap v383: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:31.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:31 smithi144 ceph-mon[51829]: pgmap v383: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:33.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:33 smithi137 ceph-mon[65321]: pgmap v384: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:33.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:33 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:33 smithi144 ceph-mon[51829]: pgmap v384: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:33.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:33 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:35.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:35 smithi137 ceph-mon[65321]: pgmap v385: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:35.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:35 smithi144 ceph-mon[51829]: pgmap v385: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:37.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:37 smithi137 ceph-mon[65321]: pgmap v386: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:37.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:37 smithi144 ceph-mon[51829]: pgmap v386: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:39.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:39 smithi137 ceph-mon[65321]: pgmap v387: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:39.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:39 smithi144 ceph-mon[51829]: pgmap v387: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:41.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:41 smithi137 ceph-mon[65321]: pgmap v388: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:41.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:41 smithi144 ceph-mon[51829]: pgmap v388: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:43.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:43 smithi137 ceph-mon[65321]: pgmap v389: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:43.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:43 smithi144 ceph-mon[51829]: pgmap v389: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:45.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:45 smithi137 ceph-mon[65321]: pgmap v390: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:45.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:45 smithi144 ceph-mon[51829]: pgmap v390: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:47.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:47 smithi137 ceph-mon[65321]: pgmap v391: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:47.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:47 smithi144 ceph-mon[51829]: pgmap v391: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:48.654 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:48 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:48.682 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch ps' 2024-09-20T23:58:48.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:48 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:58:49.628 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:49 smithi137 ceph-mon[65321]: pgmap v392: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:49.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:49 smithi144 ceph-mon[51829]: pgmap v392: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:50.596 INFO:teuthology.orchestra.run.smithi137.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:alertmanager.smithi137 smithi137 *:9093,9094 running (12m) 100s ago 21m 25.1M - 0.25.0 c8568f914cd2 3091a04827dc 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi137 smithi137 running (8m) 100s ago 21m 6895k - 19.1.1-367-g9e2ac70b cc9251b744a3 e9560489b3ff 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:crash.smithi144 smithi144 running (8m) 2m ago 20m 6903k - 19.1.1-367-g9e2ac70b cc9251b744a3 d9df82bac745 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:grafana.smithi137 smithi137 *:3000 running (108s) 100s ago 20m 41.9M - 9.4.12 2bacad6d85d8 fef4813f60ef 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi137.qhqtcp smithi137 running (2m) 100s ago 15m 37.2M - 19.1.1-367-g9e2ac70b cc9251b744a3 a7a5a9b87ed4 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mds.foofs.smithi144.ltxxzn smithi144 running (2m) 2m ago 15m 12.7M - 19.1.1-367-g9e2ac70b cc9251b744a3 a48eb0686b29 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi137.eurecx smithi137 *:8443,9283,8765 running (11m) 100s ago 22m 517M - 19.1.1-367-g9e2ac70b cc9251b744a3 852b236db413 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mgr.smithi144.ygsngz smithi144 *:8443,9283,8765 running (10m) 2m ago 20m 443M - 19.1.1-367-g9e2ac70b cc9251b744a3 27ee1a8a87c7 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi137 smithi137 running (10m) 100s ago 22m 62.1M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 a7809c1ab1cb 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:mon.smithi144 smithi144 running (9m) 2m ago 20m 47.5M 2048M 19.1.1-367-g9e2ac70b cc9251b744a3 9dda45c976fd 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.0.0.smithi137.gcqarv smithi137 *:2049 running (2m) 100s ago 14m 62.4M - 5.5 cc9251b744a3 dde28899e48b 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:nfs.foo.1.0.smithi144.kqljhb smithi144 *:2049 running (2m) 2m ago 14m 19.5M - 5.5 cc9251b744a3 dd3eeb69acf3 2024-09-20T23:58:50.597 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi137 smithi137 *:9100 running (11m) 100s ago 20m 16.6M - 1.5.0 0da6a335fe13 d0934a11af51 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:node-exporter.smithi144 smithi144 *:9100 running (11m) 2m ago 20m 17.0M - 1.5.0 0da6a335fe13 65b50f4916eb 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 smithi137 running (8m) 100s ago 19m 409M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 a5f41d92222e 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 smithi137 running (7m) 100s ago 19m 413M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 77e5507874ac 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 smithi137 running (6m) 100s ago 18m 379M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 2a94c559f5be 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 smithi137 running (5m) 100s ago 18m 397M 1936M 19.1.1-367-g9e2ac70b cc9251b744a3 c63ac3638680 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 smithi144 running (5m) 2m ago 17m 375M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 2c03e73cf0be 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 smithi144 running (4m) 2m ago 17m 351M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 a045919b60e0 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 smithi144 running (4m) 2m ago 16m 297M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 c04890e60ab3 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 smithi144 running (3m) 2m ago 16m 331M 2704M 19.1.1-367-g9e2ac70b cc9251b744a3 386ad75da8a4 2024-09-20T23:58:50.598 INFO:teuthology.orchestra.run.smithi137.stdout:prometheus.smithi137 smithi137 *:9095 running (10m) 100s ago 20m 66.8M - 2.43.0 a07b618ecd1d c7c71167efb8 2024-09-20T23:58:51.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph orch upgrade status' 2024-09-20T23:58:51.537 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:51 smithi137 ceph-mon[65321]: pgmap v393: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:51.537 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:51 smithi137 ceph-mon[65321]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:51 smithi144 ceph-mon[51829]: pgmap v393: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:51.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:51 smithi144 ceph-mon[51829]: from='client.34570 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:53.190 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "target_image": null, 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "in_progress": false, 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "which": "", 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "services_complete": [], 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "progress": null, 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "message": "", 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout: "is_paused": false 2024-09-20T23:58:53.191 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:58:53.575 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:53 smithi137 ceph-mon[65321]: pgmap v394: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:53.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:53 smithi144 ceph-mon[51829]: pgmap v394: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:53.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph health detail' 2024-09-20T23:58:54.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:54 smithi137 ceph-mon[65321]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:54.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:54 smithi144 ceph-mon[51829]: from='client.34574 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:58:55.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:55 smithi137 ceph-mon[65321]: pgmap v395: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:55.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:55 smithi144 ceph-mon[51829]: pgmap v395: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:55.848 INFO:teuthology.orchestra.run.smithi137.stdout:HEALTH_OK 2024-09-20T23:58:56.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions' 2024-09-20T23:58:56.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:56 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/2196334018' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:58:56.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:56 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/2196334018' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-09-20T23:58:57.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:57 smithi137 ceph-mon[65321]: pgmap v396: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:57.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:57 smithi144 ceph-mon[51829]: pgmap v396: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: "mon": { 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": { 2024-09-20T23:58:58.541 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": { 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 8 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "mds": { 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 2 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "overall": { 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)": 14 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:58:58.542 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-20T23:58:59.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions | jq -e '"'"'.overall | length == 1'"'"'' 2024-09-20T23:58:59.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:59 smithi137 ceph-mon[65321]: pgmap v397: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:59.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:58:59 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/734998650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:58:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:59 smithi144 ceph-mon[51829]: pgmap v397: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:58:59.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:58:59 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/734998650' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:59:01.283 INFO:teuthology.orchestra.run.smithi137.stdout:true 2024-09-20T23:59:01.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:01 smithi137 ceph-mon[65321]: pgmap v398: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:01.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:01 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3435200487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:59:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:01 smithi144 ceph-mon[51829]: pgmap v398: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:01.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:01 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3435200487' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:59:02.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -e sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 -- bash -c 'ceph versions | jq -e '"'"'.overall | keys'"'"' | grep $sha1' 2024-09-20T23:59:03.540 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:03 smithi137 ceph-mon[65321]: pgmap v399: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:03.540 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:03 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:59:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:03 smithi144 ceph-mon[51829]: pgmap v399: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:03.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:03 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-20T23:59:04.076 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)" 2024-09-20T23:59:04.653 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:04 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/725637311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:59:04.676 INFO:teuthology.task.sequential:In sequential, running task cephadm.wait_for_service... 2024-09-20T23:59:04.677 INFO:tasks.cephadm:Waiting for ceph service nfs.foo to start (timeout 300)... 2024-09-20T23:59:04.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- ceph orch ls -f json 2024-09-20T23:59:04.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:04 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/725637311' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-09-20T23:59:05.618 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:05 smithi137 ceph-mon[65321]: pgmap v400: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:05.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:05 smithi144 ceph-mon[51829]: pgmap v400: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:06.567 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-20T23:59:06.568 INFO:teuthology.orchestra.run.smithi137.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-09-20T23:37:03.647378Z", "last_refresh": "2024-09-20T23:57:10.187749Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-09-20T23:36:58.233914Z", "last_refresh": "2024-09-20T23:56:43.818328Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "spec": {"anonymous_access": true, "protocol": "https"}, "status": {"created": "2024-09-20T23:37:00.787956Z", "last_refresh": "2024-09-20T23:57:10.187900Z", "ports": [3000], "running": 1, "size": 1}}, {"placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-09-20T23:43:38.264704Z", "last_refresh": "2024-09-20T23:56:43.819283Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-09-20T23:36:56.848583Z", "last_refresh": "2024-09-20T23:56:43.818464Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi137:172.21.15.137=smithi137", "smithi144:172.21.15.144=smithi144"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-09-20T23:37:46.055672Z", "last_refresh": "2024-09-20T23:56:43.818581Z", "running": 2, "size": 2}}, {"events": ["2024-09-20T23:58:15.480876Z service:nfs.foo [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foo", "service_name": "nfs.foo", "service_type": "nfs", "spec": {"port": 2049}, "status": {"created": "2024-09-20T23:43:59.590402Z", "last_refresh": "2024-09-20T23:56:43.819391Z", "ports": [2049], "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-09-20T23:37:02.534653Z", "last_refresh": "2024-09-20T23:56:43.818692Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd", "container_image_name": "quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", "last_refresh": "2024-09-20T23:56:43.818799Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-09-20T23:36:59.385712Z", "last_refresh": "2024-09-20T23:57:10.188038Z", "ports": [9095], "running": 1, "size": 1}}] 2024-09-20T23:59:07.165 INFO:tasks.cephadm:nfs.foo has 2/2 2024-09-20T23:59:07.165 DEBUG:teuthology.parallel:result is None 2024-09-20T23:59:07.166 INFO:teuthology.run_tasks:Running task vip.exec... 2024-09-20T23:59:07.175 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:59:07.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'umount /mnt/foo' 2024-09-20T23:59:07.201 INFO:teuthology.orchestra.run.smithi137.stderr:+ umount /mnt/foo 2024-09-20T23:59:07.223 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T23:59:07.636 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:59:07.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph nfs cluster ls | grep foo' 2024-09-20T23:59:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:07 smithi144 ceph-mon[51829]: pgmap v401: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:07.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:07 smithi144 ceph-mon[51829]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:59:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:07 smithi137 ceph-mon[65321]: pgmap v401: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:07.928 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:07 smithi137 ceph-mon[65321]: from='client.34594 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-20T23:59:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "foo" 2024-09-20T23:59:09.625 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:09 smithi137 ceph-mon[65321]: pgmap v402: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:59:09.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:09 smithi144 ceph-mon[51829]: pgmap v402: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 341 B/s rd, 0 op/s 2024-09-20T23:59:10.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'ceph nfs export ls foo --detailed' 2024-09-20T23:59:10.785 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:10 smithi137 ceph-mon[65321]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:10.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:10 smithi144 ceph-mon[51829]: from='client.34598 -' entity='client.admin' cmd=[{"prefix": "nfs cluster ls", "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:11.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:11 smithi137 ceph-mon[65321]: pgmap v403: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:11.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:11 smithi144 ceph-mon[51829]: pgmap v403: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout:[ 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: "access_type": "RW", 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: "clients": [], 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_id": "foo", 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: "export_id": 1, 2024-09-20T23:59:12.173 INFO:teuthology.orchestra.run.smithi137.stdout: "fsal": { 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "fs_name": "foofs", 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "CEPH", 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "user_id": "nfs.foo.1" 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "path": "/", 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "protocols": [ 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: 4 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "pseudo": "/fake", 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "security_label": true, 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "squash": "none", 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "transports": [ 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: "TCP" 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-20T23:59:12.174 INFO:teuthology.orchestra.run.smithi137.stdout:] 2024-09-20T23:59:12.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'rados -p .nfs --all ls -' 2024-09-20T23:59:13.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:13 smithi137 ceph-mon[65321]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:13.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:13 smithi137 ceph-mon[65321]: pgmap v404: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:13.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:13 smithi137 ceph-mon[65321]: mgrmap e43: smithi137.eurecx(active, since 10m), standbys: smithi144.ygsngz 2024-09-20T23:59:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:13 smithi144 ceph-mon[51829]: from='client.34602 -' entity='client.admin' cmd=[{"prefix": "nfs export ls", "cluster_id": "foo", "detailed": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-20T23:59:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:13 smithi144 ceph-mon[51829]: pgmap v404: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:13.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:13 smithi144 ceph-mon[51829]: mgrmap e43: smithi137.eurecx(active, since 10m), standbys: smithi144.ygsngz 2024-09-20T23:59:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:foo rec-0000000000000004:nfs.foo.1 2024-09-20T23:59:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:foo grace 2024-09-20T23:59:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:foo rec-0000000000000004:nfs.foo.0 2024-09-20T23:59:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:foo conf-nfs.foo 2024-09-20T23:59:14.372 INFO:teuthology.orchestra.run.smithi137.stdout:foo export-1 2024-09-20T23:59:15.002 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-09-20T23:59:15.012 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi137.front.sepia.ceph.com 2024-09-20T23:59:15.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 -- bash -c 'set -ex 2024-09-20T23:59:15.013 DEBUG:teuthology.orchestra.run.smithi137:> [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] 2024-09-20T23:59:15.013 DEBUG:teuthology.orchestra.run.smithi137:> ' 2024-09-20T23:59:15.637 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:15 smithi137 ceph-mon[65321]: pgmap v405: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:15.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:15 smithi144 ceph-mon[51829]: pgmap v405: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:59:16.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.679 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:16 smithi137 ceph-mon[65321]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.794 INFO:teuthology.orchestra.run.smithi137.stderr:++ ceph config get mgr mgr/cephadm/migration_current 2024-09-20T23:59:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-20T23:59:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.943 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-20T23:59:16.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-20T23:59:16.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:16.944 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:16 smithi144 ceph-mon[51829]: from='mgr.24621 172.21.15.137:0/4135395612' entity='mgr.smithi137.eurecx' 2024-09-20T23:59:17.028 INFO:teuthology.orchestra.run.smithi137.stderr:+ [[ 7 -gt 2 ]] 2024-09-20T23:59:17.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:17 smithi137 ceph-mon[65321]: pgmap v406: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T23:59:17.678 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:17 smithi137 ceph-mon[65321]: from='client.? 172.21.15.137:0/3762024389' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-09-20T23:59:17.706 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-09-20T23:59:17.715 INFO:tasks.cephadm:Teardown begin 2024-09-20T23:59:17.716 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:59:17.743 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-20T23:59:17.770 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:17 smithi144 ceph-mon[51829]: pgmap v406: 97 pgs: 97 active+clean; 226 MiB data, 2.3 GiB used, 713 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2024-09-20T23:59:17.770 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:17 smithi144 ceph-mon[51829]: from='client.? 172.21.15.137:0/3762024389' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/migration_current"}]: dispatch 2024-09-20T23:59:17.776 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-09-20T23:59:17.776 DEBUG:teuthology.orchestra.run.smithi137:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T23:59:17.799 DEBUG:teuthology.orchestra.run.smithi144:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-09-20T23:59:17.835 INFO:tasks.cephadm:Stopping all daemons... 2024-09-20T23:59:17.835 INFO:tasks.cephadm.mon.smithi137:Stopping mon.smithi137... 2024-09-20T23:59:17.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137 2024-09-20T23:59:18.168 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:17 smithi137 systemd[1]: Stopping Ceph mon.smithi137 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:18.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[65294]: 2024-09-20T23:59:18.165+0000 7f5afc73e640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi137 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:18.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:18 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137[65294]: 2024-09-20T23:59:18.165+0000 7f5afc73e640 -1 mon.smithi137@0(leader) e3 *** Got Signal Terminated *** 2024-09-20T23:59:18.428 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:18 smithi137 podman[98186]: 2024-09-20 23:59:18.411168258 +0000 UTC m=+0.387903263 container died a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T23:59:19.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:18 smithi137 podman[98186]: 2024-09-20 23:59:18.795223406 +0000 UTC m=+0.771958409 container cleanup a7809c1ab1cb0735f9bfc95ef51fd3eef3ffcbd34b1b870594691f6310deb43d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-09-20T23:59:19.031 INFO:journalctl@ceph.mon.smithi137.smithi137.stdout:Sep 20 23:59:18 smithi137 bash[98186]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi137 2024-09-20T23:59:19.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi137.service' 2024-09-20T23:59:19.221 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:19.222 INFO:tasks.cephadm.mon.smithi137:Stopped mon.smithi137 2024-09-20T23:59:19.222 INFO:tasks.cephadm.mon.smithi144:Stopping mon.smithi144... 2024-09-20T23:59:19.222 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144 2024-09-20T23:59:19.667 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:19 smithi144 systemd[1]: Stopping Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:19.931 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144[51825]: 2024-09-20T23:59:19.664+0000 7fd87b5c6640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi144 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:19.931 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144[51825]: 2024-09-20T23:59:19.664+0000 7fd87b5c6640 -1 mon.smithi144@1(peon) e3 *** Got Signal Terminated *** 2024-09-20T23:59:20.193 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:19 smithi144 podman[73639]: 2024-09-20 23:59:19.930429802 +0000 UTC m=+0.515626466 container died 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-20T23:59:20.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 podman[73639]: 2024-09-20 23:59:20.456808225 +0000 UTC m=+1.042004880 container cleanup 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-20T23:59:20.693 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 bash[73639]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144 2024-09-20T23:59:20.945 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service' 2024-09-20T23:59:20.967 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 podman[73652]: 2024-09-20 23:59:20.712511576 +0000 UTC m=+0.779670360 container remove 9dda45c976fda10e168fc43e789ade6710399defad027614a3292bf2c607c85f (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-mon-smithi144, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:20.967 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service: Deactivated successfully. 2024-09-20T23:59:20.967 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 systemd[1]: Stopped Ceph mon.smithi144 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-20T23:59:20.967 INFO:journalctl@ceph.mon.smithi144.smithi144.stdout:Sep 20 23:59:20 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@mon.smithi144.service: Consumed 5.665s CPU time. 2024-09-20T23:59:20.993 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:20.993 INFO:tasks.cephadm.mon.smithi144:Stopped mon.smithi144 2024-09-20T23:59:20.993 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-20T23:59:20.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.0 2024-09-20T23:59:21.326 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:21 smithi137 systemd[1]: Stopping Ceph osd.0 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:21.678 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:21 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:59:21.324+0000 7f13aa950640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:21.678 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:21 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:59:21.324+0000 7f13aa950640 -1 osd.0 162 *** Got signal Terminated *** 2024-09-20T23:59:21.678 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:21 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0[71277]: 2024-09-20T23:59:21.324+0000 7f13aa950640 -1 osd.0 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:59:27.002 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:26 smithi137 podman[98351]: 2024-09-20 23:59:26.732327962 +0000 UTC m=+5.542241604 container died a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-09-20T23:59:27.267 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98351]: 2024-09-20 23:59:27.073537549 +0000 UTC m=+5.883451185 container cleanup a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-09-20T23:59:27.267 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 bash[98351]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0 2024-09-20T23:59:27.678 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98367]: 2024-09-20 23:59:27.266699298 +0000 UTC m=+0.528415544 container remove a5f41d92222e7395c59a99b6bae03e8d2af48a8a178417abe0d6252d53cc89aa (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:28.013 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98455]: 2024-09-20 23:59:27.594773929 +0000 UTC m=+0.016136772 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:59:28.013 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98455]: 2024-09-20 23:59:27.70112925 +0000 UTC m=+0.122492128 container create 18109a9629276de936ac438e8255d923ee6ced4e06e8adb66ccbce639c3bdc37 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731) 2024-09-20T23:59:28.013 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98455]: 2024-09-20 23:59:27.920212501 +0000 UTC m=+0.341575343 container init 18109a9629276de936ac438e8255d923ee6ced4e06e8adb66ccbce639c3bdc37 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-09-20T23:59:28.013 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:27 smithi137 podman[98455]: 2024-09-20 23:59:27.923505416 +0000 UTC m=+0.344868262 container start 18109a9629276de936ac438e8255d923ee6ced4e06e8adb66ccbce639c3bdc37 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T23:59:28.013 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:28 smithi137 podman[98455]: 2024-09-20 23:59:28.01194313 +0000 UTC m=+0.433305975 container attach 18109a9629276de936ac438e8255d923ee6ced4e06e8adb66ccbce639c3bdc37 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2) 2024-09-20T23:59:28.428 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 20 23:59:28 smithi137 podman[98455]: 2024-09-20 23:59:28.111196383 +0000 UTC m=+0.532559227 container died 18109a9629276de936ac438e8255d923ee6ced4e06e8adb66ccbce639c3bdc37 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:28.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.0.service' 2024-09-20T23:59:28.807 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:28.807 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-20T23:59:28.807 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-09-20T23:59:28.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.1 2024-09-20T23:59:29.128 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:28 smithi137 systemd[1]: Stopping Ceph osd.1 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:29.428 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:29 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:59:29.126+0000 7f6d04e6b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:29.428 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:29 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:59:29.126+0000 7f6d04e6b640 -1 osd.1 162 *** Got signal Terminated *** 2024-09-20T23:59:29.428 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:29 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1[73899]: 2024-09-20T23:59:29.126+0000 7f6d04e6b640 -1 osd.1 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:59:34.846 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:34 smithi137 podman[98557]: 2024-09-20 23:59:34.534145357 +0000 UTC m=+5.506526470 container died 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:35.145 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:34 smithi137 podman[98557]: 2024-09-20 23:59:34.926486767 +0000 UTC m=+5.898867865 container cleanup 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-09-20T23:59:35.145 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:34 smithi137 bash[98557]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1 2024-09-20T23:59:35.428 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98573]: 2024-09-20 23:59:35.144725385 +0000 UTC m=+0.613736988 container remove 77e5507874ac9c89a43029c3f09a9b7b6986ccab548b77117b4855e844dbf3a6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True) 2024-09-20T23:59:35.773 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.376436263 +0000 UTC m=+0.016718845 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:59:35.774 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.495601569 +0000 UTC m=+0.135884203 container create 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:35.774 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.698186342 +0000 UTC m=+0.338468927 container init 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-09-20T23:59:35.774 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.701925821 +0000 UTC m=+0.342208396 container start 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:35.774 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.772775144 +0000 UTC m=+0.413057732 container attach 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-09-20T23:59:36.178 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:35 smithi137 podman[98663]: 2024-09-20 23:59:35.89428618 +0000 UTC m=+0.534568763 container died 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:36.444 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 20 23:59:36 smithi137 podman[98687]: 2024-09-20 23:59:36.426484234 +0000 UTC m=+0.521500809 container remove 730ba5d9b6538cc1255fbf7e1a0606ebc5d2e6cad121f28dcbad61457fe072ef (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-1-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70) 2024-09-20T23:59:36.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.1.service' 2024-09-20T23:59:36.489 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:36.489 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-09-20T23:59:36.490 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-09-20T23:59:36.490 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.2 2024-09-20T23:59:36.822 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:36 smithi137 systemd[1]: Stopping Ceph osd.2 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:37.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:36 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:59:36.820+0000 7fab19104640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:37.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:36 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:59:36.820+0000 7fab19104640 -1 osd.2 162 *** Got signal Terminated *** 2024-09-20T23:59:37.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:36 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2[76625]: 2024-09-20T23:59:36.820+0000 7fab19104640 -1 osd.2 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:59:42.631 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:42 smithi137 podman[98766]: 2024-09-20 23:59:42.266528413 +0000 UTC m=+5.551666088 container died 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:42.914 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:42 smithi137 podman[98766]: 2024-09-20 23:59:42.703945549 +0000 UTC m=+5.989083265 container cleanup 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240731) 2024-09-20T23:59:42.914 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:42 smithi137 bash[98766]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2 2024-09-20T23:59:43.178 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:42 smithi137 podman[98816]: 2024-09-20 23:59:42.913496589 +0000 UTC m=+0.642855758 container remove 2a94c559f5be47fdbb93391ff8c5fa64c8d671a2813199efee9e8bc18e6c1363 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2) 2024-09-20T23:59:43.585 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.233199764 +0000 UTC m=+0.016184271 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:59:43.585 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.339337592 +0000 UTC m=+0.122322108 container create f0a46304a3b9709063a9bbf3f7fc7e3e5fbabc5cc3d7b5c66012fed25d33510d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:43.928 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.58431166 +0000 UTC m=+0.367296178 container init f0a46304a3b9709063a9bbf3f7fc7e3e5fbabc5cc3d7b5c66012fed25d33510d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-09-20T23:59:43.928 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.58814775 +0000 UTC m=+0.371132267 container start f0a46304a3b9709063a9bbf3f7fc7e3e5fbabc5cc3d7b5c66012fed25d33510d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-20T23:59:43.928 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.658759376 +0000 UTC m=+0.441743892 container attach f0a46304a3b9709063a9bbf3f7fc7e3e5fbabc5cc3d7b5c66012fed25d33510d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-09-20T23:59:43.928 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 20 23:59:43 smithi137 podman[98902]: 2024-09-20 23:59:43.780346412 +0000 UTC m=+0.563330928 container died f0a46304a3b9709063a9bbf3f7fc7e3e5fbabc5cc3d7b5c66012fed25d33510d (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-2-deactivate, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-09-20T23:59:44.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.2.service' 2024-09-20T23:59:44.397 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:44.397 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-09-20T23:59:44.397 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-09-20T23:59:44.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.3 2024-09-20T23:59:44.678 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:44 smithi137 systemd[1]: Stopping Ceph osd.3 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:45.178 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:59:44.739+0000 7fb7071bb640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:45.178 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:59:44.739+0000 7fb7071bb640 -1 osd.3 162 *** Got signal Terminated *** 2024-09-20T23:59:45.178 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:44 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:59:44.739+0000 7fb7071bb640 -1 osd.3 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:59:48.928 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:48 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:59:48.450+0000 7fb6ff7c9640 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:24.148368+0000 front 2024-09-20T23:59:24.148450+0000 (oldest deadline 2024-09-20T23:59:48.248483+0000) 2024-09-20T23:59:48.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:48 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:48.463+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:49.928 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:49 smithi137 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3[79213]: 2024-09-20T23:59:49.499+0000 7fb6ff7c9640 -1 osd.3 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:24.148368+0000 front 2024-09-20T23:59:24.148450+0000 (oldest deadline 2024-09-20T23:59:48.248483+0000) 2024-09-20T23:59:49.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:49 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:49.501+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:49.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:49 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:49.636+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:50.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:49 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:49.981+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:50.535 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:50 smithi137 podman[99004]: 2024-09-20 23:59:50.251774822 +0000 UTC m=+5.629975044 container died c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-20T23:59:50.867 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:50 smithi137 podman[99004]: 2024-09-20 23:59:50.640232972 +0000 UTC m=+6.018433183 container cleanup c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-20T23:59:50.867 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:50 smithi137 bash[99004]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3 2024-09-20T23:59:50.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:50 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:50.649+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:50.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:50 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:50.492+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:51.178 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:50 smithi137 podman[99021]: 2024-09-20 23:59:50.86668272 +0000 UTC m=+0.613788663 container remove c63ac363868075983f07d8263d3c0f0ed5e3b44c402982d2cba0ed0028b73f59 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:51.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:51 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:51.014+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:51.520 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.184032052 +0000 UTC m=+0.107374740 container create 5ae52d6797ef963a08f99d98689ed589bc9f33a4c887c97cf850b26b384ecfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-20T23:59:51.520 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.092650052 +0000 UTC m=+0.015992754 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:59:51.521 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.394754872 +0000 UTC m=+0.318097572 container init 5ae52d6797ef963a08f99d98689ed589bc9f33a4c887c97cf850b26b384ecfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-09-20T23:59:51.521 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.397961147 +0000 UTC m=+0.321303835 container start 5ae52d6797ef963a08f99d98689ed589bc9f33a4c887c97cf850b26b384ecfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:51.521 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.519784823 +0000 UTC m=+0.443127528 container attach 5ae52d6797ef963a08f99d98689ed589bc9f33a4c887c97cf850b26b384ecfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-20T23:59:51.857 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 20 23:59:51 smithi137 podman[99108]: 2024-09-20 23:59:51.585520764 +0000 UTC m=+0.508863461 container died 5ae52d6797ef963a08f99d98689ed589bc9f33a4c887c97cf850b26b384ecfbb (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux ) 2024-09-20T23:59:51.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:51 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:51.630+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:51.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:51 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:51.522+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:52.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.3.service' 2024-09-20T23:59:52.168 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-20T23:59:52.168 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-09-20T23:59:52.168 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-09-20T23:59:52.168 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4 2024-09-20T23:59:52.444 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:52 smithi144 systemd[1]: Stopping Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-20T23:59:52.444 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:51 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:51.985+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:52.939 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:52.469+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:52.939 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:52.476+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:52.939 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:52.458+0000 7fc6ecbd4640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-20T23:59:52.940 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:52.458+0000 7fc6ecbd4640 -1 osd.4 162 *** Got signal Terminated *** 2024-09-20T23:59:52.940 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:52.458+0000 7fc6ecbd4640 -1 osd.4 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-20T23:59:52.940 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:52.603+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:53.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:52 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:52.937+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:53.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:53 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:53.648+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:53.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:53 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:53.446+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:53.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:53 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:53.496+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:54.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:53 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:53.941+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:54.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:54 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:54.493+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:54.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:54 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:54.653+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:54.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:54 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:54.403+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:55.429 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:54 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:54.990+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:55.429 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:54 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:54.990+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-20T23:59:55.693 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:55.675+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:55.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:55.509+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:55.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:55.427+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:55.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:55.427+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-20T23:59:56.432 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:55.949+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:56.433 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:55 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:55.949+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-20T23:59:56.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:56.431+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:56.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:56.431+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-20T23:59:56.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:56.553+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:56.694 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:56.553+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-20T23:59:57.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:56.930+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:57.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:56.930+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-20T23:59:57.193 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:56.695+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:25.668396+0000 front 2024-09-20T23:59:25.668450+0000 (oldest deadline 2024-09-20T23:59:49.168241+0000) 2024-09-20T23:59:57.193 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:56 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4[56693]: 2024-09-20T23:59:56.695+0000 7fc6e51e2640 -1 osd.4 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:33.869268+0000 front 2024-09-20T23:59:33.869361+0000 (oldest deadline 2024-09-20T23:59:56.169147+0000) 2024-09-20T23:59:57.929 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:57.553+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:57.929 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:57.553+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-20T23:59:57.929 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:57.458+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:57.929 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:57.458+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-20T23:59:58.194 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:57 smithi144 podman[73828]: 2024-09-20 23:59:57.943345691 +0000 UTC m=+5.581983448 container died 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-09-20T23:59:58.194 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:57.927+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:58.194 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:57 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:57.927+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-20T23:59:58.483 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:58 smithi144 podman[73828]: 2024-09-20 23:59:58.38396649 +0000 UTC m=+6.022604290 container cleanup 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-20T23:59:58.483 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:58 smithi144 bash[73828]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4 2024-09-20T23:59:58.943 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:58 smithi144 podman[73843]: 2024-09-20 23:59:58.585529688 +0000 UTC m=+0.633182983 container remove 2c03e73cf0be99c44ac167a964b2cf15790708761705161df82b36ad2035cee6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-20T23:59:58.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:58.558+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:58.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:58.558+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-20T23:59:58.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:58.481+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:58.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:58.481+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-20T23:59:59.285 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:58.945+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-20T23:59:59.285 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:58 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:58.945+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-20T23:59:59.285 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:58 smithi144 podman[73932]: 2024-09-20 23:59:58.899605102 +0000 UTC m=+0.015724550 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-20T23:59:59.285 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:59 smithi144 podman[73932]: 2024-09-20 23:59:59.031733671 +0000 UTC m=+0.147853150 container create ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-20T23:59:59.285 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:59 smithi144 podman[73932]: 2024-09-20 23:59:59.284253977 +0000 UTC m=+0.400373410 container init ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-20T23:59:59.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:59.531+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-20T23:59:59.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-20T23:59:59.531+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-20T23:59:59.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:59.510+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-20T23:59:59.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-20T23:59:59.510+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-20T23:59:59.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:59 smithi144 podman[73932]: 2024-09-20 23:59:59.287910811 +0000 UTC m=+0.404030245 container start ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-20T23:59:59.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:59 smithi144 podman[73932]: 2024-09-20 23:59:59.392791189 +0000 UTC m=+0.508910624 container attach ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-20T23:59:59.694 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 20 23:59:59 smithi144 podman[73932]: 2024-09-20 23:59:59.485182628 +0000 UTC m=+0.601302074 container died ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:00:00.047 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service' 2024-09-21T00:00:00.070 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 21 00:00:00 smithi144 podman[73955]: 2024-09-21 00:00:00.013377879 +0000 UTC m=+0.522216124 container remove ccebc7fce30093e34e0473a08822ab4ba6f9935d98bb23014911b051b5127368 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-4-deactivate, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-09-21T00:00:00.070 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 21 00:00:00 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service: Deactivated successfully. 2024-09-21T00:00:00.071 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 21 00:00:00 smithi144 systemd[1]: Stopped Ceph osd.4 for 1150c2f6-77a9-11ef-baf5-efdc52797490. 2024-09-21T00:00:00.071 INFO:journalctl@ceph.osd.4.smithi144.stdout:Sep 21 00:00:00 smithi144 systemd[1]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.4.service: Consumed 1min 7.725s CPU time. 2024-09-21T00:00:00.071 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:59.928+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:00.071 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 20 23:59:59 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-20T23:59:59.928+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:00.086 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T00:00:00.087 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-09-21T00:00:00.087 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-09-21T00:00:00.087 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.5 2024-09-21T00:00:00.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 systemd[1]: Stopping Ceph osd.5 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-21T00:00:00.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:00.452+0000 7f8055bc1640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T00:00:00.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:00.452+0000 7f8055bc1640 -1 osd.5 162 *** Got signal Terminated *** 2024-09-21T00:00:00.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:00.452+0000 7f8055bc1640 -1 osd.5 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T00:00:00.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:00.496+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-21T00:00:00.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:00.496+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-21T00:00:00.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:00.480+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:00.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:00.480+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:01.256 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:00.977+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:01.256 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:00 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:00.977+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:01.531 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:01.485+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:01.531 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:01.485+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:01.937 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:01.529+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-21T00:00:01.937 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:01.529+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-21T00:00:02.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:01.935+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:02.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:01 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:01.935+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:02.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:02.554+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-21T00:00:02.693 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:02.554+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-21T00:00:02.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:02.489+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:02.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:02.489+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:03.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:02.949+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:03.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:02 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:02.949+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:03.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:03.479+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:03.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:03.479+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:03.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:03.528+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-21T00:00:03.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:03.528+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-21T00:00:03.944 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:03.528+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.465077+0000 front 2024-09-20T23:59:41.465114+0000 (oldest deadline 2024-09-21T00:00:03.165053+0000) 2024-09-21T00:00:04.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:03.992+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:04.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:03 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:03.992+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:04.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:04.529+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:22.861964+0000 front 2024-09-20T23:59:22.862038+0000 (oldest deadline 2024-09-20T23:59:51.662372+0000) 2024-09-21T00:00:04.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:04.529+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:31.663123+0000 front 2024-09-20T23:59:31.662940+0000 (oldest deadline 2024-09-20T23:59:55.763056+0000) 2024-09-21T00:00:04.943 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5[58947]: 2024-09-21T00:00:04.529+0000 7f804e1cf640 -1 osd.5 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.465077+0000 front 2024-09-20T23:59:41.465114+0000 (oldest deadline 2024-09-21T00:00:03.165053+0000) 2024-09-21T00:00:04.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:04.468+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:04.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:04.469+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:04.944 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:04.469+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:05.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:04.985+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:05.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:04.985+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:05.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:04 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:04.985+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:05.874 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:05 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:05.468+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:05.874 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:05 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:05.468+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:05.874 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:05 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:05.468+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:06.193 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:05 smithi144 podman[74074]: 2024-09-21 00:00:05.873368827 +0000 UTC m=+5.542373726 container died a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T00:00:06.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:06.019+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:06.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:06.019+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:06.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:06.019+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:06.490 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:06 smithi144 podman[74074]: 2024-09-21 00:00:06.193625876 +0000 UTC m=+5.862630769 container cleanup a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2) 2024-09-21T00:00:06.490 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:06 smithi144 bash[74074]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5 2024-09-21T00:00:06.490 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:06 smithi144 podman[74102]: 2024-09-21 00:00:06.389684409 +0000 UTC m=+0.512169941 container remove a045919b60e0128bc68e4fd4cc05603ac39715cf2c1134570d9dbbc88627e4a2 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T00:00:06.809 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:06.488+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:06.810 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:06.488+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:06.810 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:06 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:06.488+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:07.121 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:07.055+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:07.121 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:07.055+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:07.121 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:07.055+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:07.121 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:06 smithi144 podman[74193]: 2024-09-21 00:00:06.711581505 +0000 UTC m=+0.015607126 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T00:00:07.121 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:06 smithi144 podman[74193]: 2024-09-21 00:00:06.826533518 +0000 UTC m=+0.130559185 container create 875fdd6f3b21e05cdfcf47a38bfcc32823795ab8a31b6167e25a200290e3717e (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-09-21T00:00:07.122 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:07 smithi144 podman[74193]: 2024-09-21 00:00:07.037331733 +0000 UTC m=+0.341357355 container init 875fdd6f3b21e05cdfcf47a38bfcc32823795ab8a31b6167e25a200290e3717e (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, ceph=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=) 2024-09-21T00:00:07.122 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:07 smithi144 podman[74193]: 2024-09-21 00:00:07.040750273 +0000 UTC m=+0.344775895 container start 875fdd6f3b21e05cdfcf47a38bfcc32823795ab8a31b6167e25a200290e3717e (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-09-21T00:00:07.122 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:07 smithi144 podman[74193]: 2024-09-21 00:00:07.120529471 +0000 UTC m=+0.424555108 container attach 875fdd6f3b21e05cdfcf47a38bfcc32823795ab8a31b6167e25a200290e3717e (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-09-21T00:00:07.443 INFO:journalctl@ceph.osd.5.smithi144.stdout:Sep 21 00:00:07 smithi144 podman[74193]: 2024-09-21 00:00:07.237309646 +0000 UTC m=+0.541335279 container died 875fdd6f3b21e05cdfcf47a38bfcc32823795ab8a31b6167e25a200290e3717e (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T00:00:07.775 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:07.462+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:07.775 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:07.462+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:07.775 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:07 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:07.462+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:07.793 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.5.service' 2024-09-21T00:00:07.832 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T00:00:07.832 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-09-21T00:00:07.832 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-09-21T00:00:07.833 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.6 2024-09-21T00:00:08.056 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:07 smithi144 systemd[1]: Stopping Ceph osd.6 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-21T00:00:08.056 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.054+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:08.056 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.054+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:08.056 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.054+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:08.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.169+0000 7f569c6e3640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T00:00:08.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.169+0000 7f569c6e3640 -1 osd.6 162 *** Got signal Terminated *** 2024-09-21T00:00:08.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:08.169+0000 7f569c6e3640 -1 osd.6 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T00:00:08.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:08.484+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:08.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:08.484+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:08.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:08 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:08.484+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:09.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:09.063+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:09.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:09.063+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:09.443 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:09.063+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:09.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:09.452+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:09.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:09.452+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:09.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:09 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:09.452+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:10.425 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:10.045+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:10.425 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:10.045+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:10.425 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:10.045+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:10.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:10.423+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:10.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:10.423+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:10.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:10 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:10.423+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:11.433 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:11.093+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:11.433 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:11.093+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:11.434 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:11.093+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:11.434 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:11.093+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:47.113145+0000 front 2024-09-20T23:59:47.113180+0000 (oldest deadline 2024-09-21T00:00:10.513551+0000) 2024-09-21T00:00:11.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:11.432+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:11.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:11.432+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:11.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:11 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:11.432+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:12.423 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:12.114+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:12.423 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:12.114+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:12.423 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:12.114+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:12.423 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:12.114+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:47.113145+0000 front 2024-09-20T23:59:47.113180+0000 (oldest deadline 2024-09-21T00:00:10.513551+0000) 2024-09-21T00:00:12.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:12.421+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:12.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:12.421+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:12.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:12.421+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:12.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:12 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:12.421+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:13.404 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:13.083+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:23.809436+0000 front 2024-09-20T23:59:23.809627+0000 (oldest deadline 2024-09-20T23:59:49.709533+0000) 2024-09-21T00:00:13.405 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:13.083+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:32.610688+0000 front 2024-09-20T23:59:32.610593+0000 (oldest deadline 2024-09-20T23:59:54.910721+0000) 2024-09-21T00:00:13.405 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:13.083+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:39.011778+0000 front 2024-09-20T23:59:39.011726+0000 (oldest deadline 2024-09-21T00:00:04.311680+0000) 2024-09-21T00:00:13.405 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6[61219]: 2024-09-21T00:00:13.083+0000 7f5698cf9640 -1 osd.6 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:47.113145+0000 front 2024-09-20T23:59:47.113180+0000 (oldest deadline 2024-09-21T00:00:10.513551+0000) 2024-09-21T00:00:13.693 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 podman[74296]: 2024-09-21 00:00:13.553199143 +0000 UTC m=+5.502603081 container died c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T00:00:13.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:13.403+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:13.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:13.403+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:13.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:13.403+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:13.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:13 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:13.403+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:14.104 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 podman[74296]: 2024-09-21 00:00:13.897401332 +0000 UTC m=+5.846805259 container cleanup c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:00:14.104 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:13 smithi144 bash[74296]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6 2024-09-21T00:00:14.358 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74311]: 2024-09-21 00:00:14.103754488 +0000 UTC m=+0.547489075 container remove c04890e60ab3ea80127cb9591c0d6a2c6835f782b45657d60687d9c5f65d4f45 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T00:00:14.624 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.505609694 +0000 UTC m=+0.109447412 container create 83634a6340c818fcf9ca6836875ff5854f2915405ec34fa2f13b57409e95f836 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-09-21T00:00:14.625 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.411638696 +0000 UTC m=+0.015476377 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T00:00:14.625 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:14 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:14.356+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:14.625 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:14 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:14.356+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:14.625 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:14 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:14.356+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:14.625 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:14 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:14.356+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:15.193 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.70270414 +0000 UTC m=+0.306541812 container init 83634a6340c818fcf9ca6836875ff5854f2915405ec34fa2f13b57409e95f836 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=squid-9e2ac70, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T00:00:15.194 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.705883746 +0000 UTC m=+0.309721421 container start 83634a6340c818fcf9ca6836875ff5854f2915405ec34fa2f13b57409e95f836 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, io.buildah.version=1.37.2) 2024-09-21T00:00:15.194 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.786935167 +0000 UTC m=+0.390772840 container attach 83634a6340c818fcf9ca6836875ff5854f2915405ec34fa2f13b57409e95f836 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-09-21T00:00:15.194 INFO:journalctl@ceph.osd.6.smithi144.stdout:Sep 21 00:00:14 smithi144 podman[74402]: 2024-09-21 00:00:14.892221413 +0000 UTC m=+0.496059098 container died 83634a6340c818fcf9ca6836875ff5854f2915405ec34fa2f13b57409e95f836 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-6-deactivate, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T00:00:15.609 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.401+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:15.609 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.401+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:15.609 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.401+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:15.609 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.401+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:15.631 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.6.service' 2024-09-21T00:00:15.670 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T00:00:15.670 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-09-21T00:00:15.670 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T00:00:15.670 DEBUG:teuthology.orchestra.run.smithi144:> sudo systemctl stop ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.7 2024-09-21T00:00:15.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 systemd[1]: Stopping Ceph osd.7 for 1150c2f6-77a9-11ef-baf5-efdc52797490... 2024-09-21T00:00:16.436 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.994+0000 7fa801ea0640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T00:00:16.436 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.994+0000 7fa801ea0640 -1 osd.7 162 *** Got signal Terminated *** 2024-09-21T00:00:16.436 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:15 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:15.994+0000 7fa801ea0640 -1 osd.7 162 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T00:00:16.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:16 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:16.435+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:16.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:16 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:16.435+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:16.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:16 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:16.435+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:16.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:16 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:16.435+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:17.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:17 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:17.465+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:17.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:17 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:17.465+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:17.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:17 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:17.465+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:17.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:17 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:17.465+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:18.844 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:18 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:18.454+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:18.845 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:18 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:18.455+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:18.845 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:18 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:18.455+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:18.845 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:18 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:18.455+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:19.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:19.467+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:19.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:19.467+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:19.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:19.467+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:19.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:19.467+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:19.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:19 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:19.467+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.144:6804 osd.4 since back 2024-09-20T23:59:57.429811+0000 front 2024-09-20T23:59:57.429960+0000 (oldest deadline 2024-09-21T00:00:19.129910+0000) 2024-09-21T00:00:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:20.419+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6806 osd.0 since back 2024-09-20T23:59:20.725237+0000 front 2024-09-20T23:59:20.725224+0000 (oldest deadline 2024-09-20T23:59:47.725439+0000) 2024-09-21T00:00:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:20.419+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6814 osd.1 since back 2024-09-20T23:59:29.426382+0000 front 2024-09-20T23:59:29.426343+0000 (oldest deadline 2024-09-20T23:59:55.326253+0000) 2024-09-21T00:00:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:20.419+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6822 osd.2 since back 2024-09-20T23:59:41.127789+0000 front 2024-09-20T23:59:41.127703+0000 (oldest deadline 2024-09-21T00:00:04.027608+0000) 2024-09-21T00:00:20.693 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:20.419+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.137:6830 osd.3 since back 2024-09-20T23:59:46.229115+0000 front 2024-09-20T23:59:46.229043+0000 (oldest deadline 2024-09-21T00:00:12.129014+0000) 2024-09-21T00:00:20.694 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:20 smithi144 ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7[63465]: 2024-09-21T00:00:20.419+0000 7fa7fe4b6640 -1 osd.7 162 heartbeat_check: no reply from 172.21.15.144:6804 osd.4 since back 2024-09-20T23:59:57.429811+0000 front 2024-09-20T23:59:57.429960+0000 (oldest deadline 2024-09-21T00:00:19.129910+0000) 2024-09-21T00:00:21.688 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:21 smithi144 podman[74505]: 2024-09-21 00:00:21.418661351 +0000 UTC m=+5.528336225 container died 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T00:00:21.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:21 smithi144 podman[74505]: 2024-09-21 00:00:21.852524576 +0000 UTC m=+5.962199442 container cleanup 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, ceph=True, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:00:21.943 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:21 smithi144 bash[74505]: ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7 2024-09-21T00:00:22.443 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74565]: 2024-09-21 00:00:22.053909107 +0000 UTC m=+0.633757296 container remove 386ad75da8a4b8429a43e2c10cacb0c10eec315512553ae65cdbac4dc5d1fed6 (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7, ceph=True, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:00:22.753 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.366079347 +0000 UTC m=+0.015784215 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T00:00:22.753 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.465529955 +0000 UTC m=+0.115234850 container create c7b94fc66917415ffbb49ab502ee139b2747c636cbb38537e6c5134454685d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD) 2024-09-21T00:00:22.753 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.667943102 +0000 UTC m=+0.317647965 container init c7b94fc66917415ffbb49ab502ee139b2747c636cbb38537e6c5134454685d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T00:00:22.753 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.671236094 +0000 UTC m=+0.320940954 container start c7b94fc66917415ffbb49ab502ee139b2747c636cbb38537e6c5134454685d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-09-21T00:00:22.753 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.751114196 +0000 UTC m=+0.400819065 container attach c7b94fc66917415ffbb49ab502ee139b2747c636cbb38537e6c5134454685d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T00:00:23.147 INFO:journalctl@ceph.osd.7.smithi144.stdout:Sep 21 00:00:22 smithi144 podman[74653]: 2024-09-21 00:00:22.859179408 +0000 UTC m=+0.508884264 container died c7b94fc66917415ffbb49ab502ee139b2747c636cbb38537e6c5134454685d4a (image=quay.ceph.io/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-1150c2f6-77a9-11ef-baf5-efdc52797490-osd-7-deactivate, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-09-21T00:00:23.453 DEBUG:teuthology.orchestra.run.smithi144:> sudo pkill -f 'journalctl -f -n 0 -u ceph-1150c2f6-77a9-11ef-baf5-efdc52797490@osd.7.service' 2024-09-21T00:00:23.495 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T00:00:23.496 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T00:00:23.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T00:03:22.463 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 --force --keep-logs 2024-09-21T00:03:43.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T00:03:43.719 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-09-21T00:03:43.746 INFO:tasks.cephadm:Archiving crash dumps... 2024-09-21T00:03:43.746 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi137/crash 2024-09-21T00:03:43.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash -- . 2024-09-21T00:03:43.796 INFO:teuthology.orchestra.run.smithi137.stderr:tar: /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T00:03:43.796 INFO:teuthology.orchestra.run.smithi137.stderr:tar: Error is not recoverable: exiting now 2024-09-21T00:03:43.798 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi144/crash 2024-09-21T00:03:43.799 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash -- . 2024-09-21T00:03:43.864 INFO:teuthology.orchestra.run.smithi144.stderr:tar: /var/lib/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/crash: Cannot open: No such file or directory 2024-09-21T00:03:43.864 INFO:teuthology.orchestra.run.smithi144.stderr:tar: Error is not recoverable: exiting now 2024-09-21T00:03:43.865 INFO:tasks.cephadm:Checking cluster log for badness... 2024-09-21T00:03:43.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log | egrep CEPHADM_ | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v CEPHADM_REFRESH_FAILED | head -n 1 2024-09-21T00:03:43.895 INFO:tasks.cephadm:Compressing logs... 2024-09-21T00:03:43.896 DEBUG:teuthology.orchestra.run.smithi137:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T00:03:43.939 DEBUG:teuthology.orchestra.run.smithi144:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-21T00:03:43.962 INFO:teuthology.orchestra.run.smithi137.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.0.log 2024-09-21T00:03:43.962 INFO:teuthology.orchestra.run.smithi137.stderr:: No such file or directory 2024-09-21T00:03:43.962 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.1.log 2024-09-21T00:03:43.966 INFO:teuthology.orchestra.run.smithi144.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi144.log 2024-09-21T00:03:43.966 INFO:teuthology.orchestra.run.smithi144.stderr:: No such file or directory 2024-09-21T00:03:43.966 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.7.log 2024-09-21T00:03:43.966 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi144.log: /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.4.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi144.ygsngz.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.4.log: /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi144.ygsngz.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi144.ltxxzn.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log: /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi144.ltxxzn.log: 85.7% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.6.log 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr: 88.1% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi144.ltxxzn.log.gz 2024-09-21T00:03:43.967 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.5.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi137.eurecx.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.0.log: /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi137.eurecx.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.2.log 2024-09-21T00:03:43.968 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi137.qhqtcp.log 2024-09-21T00:03:43.969 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log 2024-09-21T00:03:43.969 INFO:teuthology.orchestra.run.smithi137.stderr: 85.7% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.cephadm.log.gz 2024-09-21T00:03:43.969 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi137.qhqtcp.log: 91.6% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T00:03:43.969 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log 2024-09-21T00:03:43.969 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.5.log: 91.7% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.audit.log.gz 2024-09-21T00:03:43.970 INFO:teuthology.orchestra.run.smithi144.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T00:03:43.973 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log: 87.0% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T00:03:43.975 INFO:teuthology.orchestra.run.smithi137.stderr: 91.2% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T00:03:43.977 INFO:teuthology.orchestra.run.smithi144.stderr:/var/log/ceph/cephadm.log: 91.2% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-volume.log.gz 2024-09-21T00:03:43.978 INFO:teuthology.orchestra.run.smithi137.stderr:gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.3.log 2024-09-21T00:03:43.979 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log: gzip -5 --verbose -- /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi137.log 2024-09-21T00:03:43.979 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-09-21T00:03:43.981 INFO:teuthology.orchestra.run.smithi144.stderr: 90.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T00:03:43.982 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi137.log: 87.0% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph.log.gz 2024-09-21T00:03:43.990 INFO:teuthology.orchestra.run.smithi137.stderr:/var/log/ceph/cephadm.log: 91.3% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mds.foofs.smithi137.qhqtcp.log.gz 2024-09-21T00:03:44.020 INFO:teuthology.orchestra.run.smithi137.stderr: 91.9% -- replaced with /var/log/ceph/cephadm.log.gz 2024-09-21T00:03:44.033 INFO:teuthology.orchestra.run.smithi144.stderr: 89.8% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi144.ygsngz.log.gz 2024-09-21T00:03:44.339 INFO:teuthology.orchestra.run.smithi137.stderr: 88.6% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mgr.smithi137.eurecx.log.gz 2024-09-21T00:03:44.414 INFO:teuthology.orchestra.run.smithi144.stderr: 92.6% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi144.log.gz 2024-09-21T00:03:45.252 INFO:teuthology.orchestra.run.smithi137.stderr: 91.9% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-mon.smithi137.log.gz 2024-09-21T00:03:55.921 INFO:teuthology.orchestra.run.smithi137.stderr: 94.5% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.2.log.gz 2024-09-21T00:03:58.913 INFO:teuthology.orchestra.run.smithi144.stderr: 94.2% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.6.log.gz 2024-09-21T00:04:00.104 INFO:teuthology.orchestra.run.smithi144.stderr: 94.0% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.4.log.gz 2024-09-21T00:04:00.446 INFO:teuthology.orchestra.run.smithi137.stderr: 94.1% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.0.log.gz 2024-09-21T00:04:01.827 INFO:teuthology.orchestra.run.smithi137.stderr: 94.2% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.1.log.gz 2024-09-21T00:04:02.511 INFO:teuthology.orchestra.run.smithi137.stderr: 94.0% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.3.log.gz 2024-09-21T00:04:02.513 INFO:teuthology.orchestra.run.smithi137.stderr: 2024-09-21T00:04:02.513 INFO:teuthology.orchestra.run.smithi137.stderr:real 0m18.561s 2024-09-21T00:04:02.513 INFO:teuthology.orchestra.run.smithi137.stderr:user 1m3.911s 2024-09-21T00:04:02.513 INFO:teuthology.orchestra.run.smithi137.stderr:sys 0m2.548s 2024-09-21T00:04:03.481 INFO:teuthology.orchestra.run.smithi144.stderr: 94.2% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.5.log.gz 2024-09-21T00:04:06.862 INFO:teuthology.orchestra.run.smithi144.stderr: 94.0% -- replaced with /var/log/ceph/1150c2f6-77a9-11ef-baf5-efdc52797490/ceph-osd.7.log.gz 2024-09-21T00:04:06.864 INFO:teuthology.orchestra.run.smithi144.stderr: 2024-09-21T00:04:06.864 INFO:teuthology.orchestra.run.smithi144.stderr:real 0m22.911s 2024-09-21T00:04:06.864 INFO:teuthology.orchestra.run.smithi144.stderr:user 1m11.019s 2024-09-21T00:04:06.864 INFO:teuthology.orchestra.run.smithi144.stderr:sys 0m2.839s 2024-09-21T00:04:06.864 INFO:tasks.cephadm:Archiving logs... 2024-09-21T00:04:06.864 DEBUG:teuthology.misc:Transferring archived files from smithi137:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi137/log 2024-09-21T00:04:06.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T00:04:15.541 DEBUG:teuthology.misc:Transferring archived files from smithi144:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi144/log 2024-09-21T00:04:15.542 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-21T00:04:26.052 INFO:tasks.cephadm:Removing cluster... 2024-09-21T00:04:26.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 --force 2024-09-21T00:04:26.413 DEBUG:teuthology.orchestra.run.smithi144:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 1150c2f6-77a9-11ef-baf5-efdc52797490 --force 2024-09-21T00:04:26.791 INFO:tasks.cephadm:Removing cephadm ... 2024-09-21T00:04:26.791 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T00:04:26.807 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf /home/ubuntu/cephtest/cephadm 2024-09-21T00:04:26.822 INFO:tasks.cephadm:Teardown complete 2024-09-21T00:04:26.822 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-21T00:04:26.831 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-21T00:04:26.831 DEBUG:teuthology.orchestra.run.smithi137:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T00:04:26.849 DEBUG:teuthology.orchestra.run.smithi144:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T00:04:26.861 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:^* hv01.front.sepia.ceph.com 3 6 377 27 +76us[ +120us] +/- 53ms 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 30 -1166us[-1123us] +/- 69ms 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 31 +1035us[+1078us] +/- 65ms 2024-09-21T00:04:26.866 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:04:26.876 INFO:teuthology.orchestra.run.smithi144.stderr:bash: line 1: ntpq: command not found 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:=============================================================================== 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:^* hv01.front.sepia.ceph.com 3 6 377 28 +84us[ +120us] +/- 53ms 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 34 -1176us[-1141us] +/- 69ms 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 29 +1022us[+1058us] +/- 65ms 2024-09-21T00:04:26.881 INFO:teuthology.orchestra.run.smithi144.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T00:04:26.882 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-21T00:04:27.304 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-21T00:04:27.305 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-21T00:04:28.919 DEBUG:teuthology.orchestra.run.smithi137:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T00:04:29.004 DEBUG:teuthology.orchestra.run.smithi144:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-09-21T00:04:29.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T00:04:29.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:04:29.116 DEBUG:teuthology.orchestra.run.smithi144:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-09-21T00:04:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:04:29.146 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-21T00:04:29.158 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-21T00:04:30.065 INFO:teuthology.task.internal:Duration was 1944.324649 seconds 2024-09-21T00:04:30.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-21T00:04:31.016 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-21T00:04:31.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T00:04:31.019 DEBUG:teuthology.orchestra.run.smithi144:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-21T00:04:31.063 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T00:04:31.065 INFO:teuthology.orchestra.run.smithi144.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T00:04:31.487 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-21T00:04:31.487 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi137.front.sepia.ceph.com 2024-09-21T00:04:31.487 DEBUG:teuthology.orchestra.run.smithi137:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T00:04:31.510 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi144.front.sepia.ceph.com 2024-09-21T00:04:31.510 DEBUG:teuthology.orchestra.run.smithi144:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-21T00:04:31.539 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-21T00:04:31.539 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T00:04:31.552 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-21T00:04:31.609 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-21T00:04:31.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T00:04:31.623 DEBUG:teuthology.orchestra.run.smithi144:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-21T00:04:45.429 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-21T00:04:45.506 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-21T00:04:45.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T00:04:45.532 DEBUG:teuthology.orchestra.run.smithi144:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-21T00:04:45.563 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-21T00:04:46.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T00:04:46.381 DEBUG:teuthology.orchestra.run.smithi144:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-21T00:04:46.407 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = core 2024-09-21T00:04:46.409 INFO:teuthology.orchestra.run.smithi144.stdout:kernel.core_pattern = core 2024-09-21T00:04:46.545 DEBUG:teuthology.orchestra.run.smithi137:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T00:04:46.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:04:46.603 DEBUG:teuthology.orchestra.run.smithi144:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-21T00:04:46.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T00:04:46.621 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-21T00:04:47.513 INFO:teuthology.task.internal:Transferring archived files... 2024-09-21T00:04:47.516 DEBUG:teuthology.misc:Transferring archived files from smithi137:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi137 2024-09-21T00:04:47.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T00:04:47.583 DEBUG:teuthology.misc:Transferring archived files from smithi144:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913089/remote/smithi144 2024-09-21T00:04:47.584 DEBUG:teuthology.orchestra.run.smithi144:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-21T00:04:47.626 INFO:teuthology.task.internal:Removing archive directory... 2024-09-21T00:04:47.626 DEBUG:teuthology.orchestra.run.smithi137:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T00:04:47.629 DEBUG:teuthology.orchestra.run.smithi144:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-21T00:04:47.679 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-21T00:04:48.456 INFO:teuthology.task.internal:Not uploading archives. 2024-09-21T00:04:48.456 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-21T00:04:49.618 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-21T00:04:49.618 DEBUG:teuthology.orchestra.run.smithi137:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T00:04:49.621 DEBUG:teuthology.orchestra.run.smithi144:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-21T00:04:49.637 INFO:teuthology.orchestra.run.smithi137.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 00:04 /home/ubuntu/cephtest 2024-09-21T00:04:49.640 INFO:teuthology.orchestra.run.smithi144.stdout: 262169 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 21 00:04 /home/ubuntu/cephtest 2024-09-21T00:04:49.641 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-21T00:04:50.518 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-21T00:04:52.165 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 1944.3246486186981 owner: scheduled_teuthology@teuthology success: true 2024-09-21T00:04:52.165 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T00:04:52.255 INFO:teuthology.run:pass