2024-04-07T13:52:08.403 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-04-07T13:52:08.403 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187 --owner scheduled_teuthology@teuthology --name teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi --description orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} --verbose -- /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/orig.config.yaml 2024-04-07T13:52:08.439 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-07T13:52:08.520 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187 branch: main description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} email: ceph-qa@ceph.com first_in_suite: false job_id: '7634187' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: main sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d owner: scheduled_teuthology@teuthology priority: 950 repo: https://git.ceph.com/ceph.git roles: - - host.a - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - host.b - osd.4 - osd.5 - osd.6 - osd.7 seed: 7151 sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d sleep_before_teardown: 0 subset: 39/64 suite: orch suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d targets: smithi111.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXgwrwNSKGT7cIlN4zgFeJ2gcVfHjH+x7d72ytA5WhkIrhcWKLrSh033S9hw/RA6YjPdofi4DGFrdisK3rGXPw= smithi145.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMaXteOlXjxMKd/pO4HJZvN7j53JHeyHukDvNFO3KA2Yi1tjH7Vf2qhr+59SY1gOG/+0oFi5qqciqug6vqo7dA= tasks: - cephadm: cephadm_branch: v17.2.0 cephadm_git_url: https://github.com/ceph/ceph image: quay.io/ceph/ceph:v17.2.0 roleless: true - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - vip.exec: all-hosts: - systemctl stop nfs-server - cephadm.shell: host.a: - ceph fs volume create foofs - cephadm.wait_for_service: service: mds.foofs - cephadm.shell: host.a: - ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2 - ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake - while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done - vip.exec: host.a: - mkdir /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done - echo test > /mnt/foo/testfile - sync - parallel: - upgrade-tasks - workload-tasks - vip.exec: host.a: - umount /mnt/foo - cephadm.shell: host.a: - ceph nfs cluster ls | grep foo - ceph nfs export ls foo --detailed - rados -p .nfs --all ls - - cephadm.shell: host.a: - 'set -ex [[ `ceph config get mgr mgr/cephadm/migration_current` -gt 2 ]] ' teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-04-01_20:08:13 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph mgr module enable nfs --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 - cephadm.wait_for_service: service: nfs.foo user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - exec: host.a: - cd /mnt/foo && dbench 5 -t 600 || true - umount /mnt/foo - while ! mount -t nfs $(hostname):/fake /mnt/foo ; do sleep 5 ; done - cd /mnt/foo && dbench 5 -t 5 2024-04-07T13:52:08.520 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa; will attempt to use it 2024-04-07T13:52:08.521 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks 2024-04-07T13:52:08.521 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-07T13:52:08.523 INFO:teuthology.task.internal:Checking packages... 2024-04-07T13:52:08.545 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'e5c885fe95ee982ff7b45214daa61b1a6d09d57d' 2024-04-07T13:52:08.545 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-07T13:52:08.545 INFO:teuthology.packaging:ref: None 2024-04-07T13:52:08.545 INFO:teuthology.packaging:tag: None 2024-04-07T13:52:08.545 INFO:teuthology.packaging:branch: main 2024-04-07T13:52:08.545 INFO:teuthology.packaging:sha1: e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T13:52:08.545 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-04-07T13:52:08.906 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2778.ga971fd5f 2024-04-07T13:52:08.908 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-07T13:52:08.915 INFO:teuthology.task.internal:no buildpackages task found 2024-04-07T13:52:08.916 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-07T13:52:08.925 INFO:teuthology.task.internal:Saving configuration 2024-04-07T13:52:08.939 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-07T13:52:08.945 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T13:52:08.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 13:46:50.584020', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXgwrwNSKGT7cIlN4zgFeJ2gcVfHjH+x7d72ytA5WhkIrhcWKLrSh033S9hw/RA6YjPdofi4DGFrdisK3rGXPw='} 2024-04-07T13:52:08.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 13:46:50.582692', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMaXteOlXjxMKd/pO4HJZvN7j53JHeyHukDvNFO3KA2Yi1tjH7Vf2qhr+59SY1gOG/+0oFi5qqciqug6vqo7dA='} 2024-04-07T13:52:08.994 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-07T13:52:09.000 INFO:teuthology.task.internal:roles: ubuntu@smithi111.front.sepia.ceph.com - ['host.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-04-07T13:52:09.000 INFO:teuthology.task.internal:roles: ubuntu@smithi145.front.sepia.ceph.com - ['host.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-04-07T13:52:09.000 INFO:teuthology.run_tasks:Running task console_log... 2024-04-07T13:52:09.072 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6bc4d21f70>, signals=[15]) 2024-04-07T13:52:09.073 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-07T13:52:09.080 INFO:teuthology.task.internal:Opening connections... 2024-04-07T13:52:09.080 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T13:52:09.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:52:09.171 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi145.front.sepia.ceph.com 2024-04-07T13:52:09.172 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:52:09.260 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-07T13:52:09.267 DEBUG:teuthology.orchestra.run.smithi111:> uname -m 2024-04-07T13:52:09.284 INFO:teuthology.orchestra.run.smithi111.stdout:x86_64 2024-04-07T13:52:09.284 DEBUG:teuthology.orchestra.run.smithi111:> cat /etc/os-release 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:NAME="CentOS Stream" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION="9" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:ID="centos" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:ID_LIKE="rhel fedora" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:VERSION_ID="9" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:PLATFORM_ID="platform:el9" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:ANSI_COLOR="0;31" 2024-04-07T13:52:09.341 INFO:teuthology.orchestra.run.smithi111.stdout:LOGO="fedora-logo-icon" 2024-04-07T13:52:09.342 INFO:teuthology.orchestra.run.smithi111.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-07T13:52:09.342 INFO:teuthology.orchestra.run.smithi111.stdout:HOME_URL="https://centos.org/" 2024-04-07T13:52:09.342 INFO:teuthology.orchestra.run.smithi111.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-07T13:52:09.342 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-07T13:52:09.342 INFO:teuthology.orchestra.run.smithi111.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-07T13:52:09.342 INFO:teuthology.lock.ops:Updating smithi111.front.sepia.ceph.com on lock server 2024-04-07T13:52:09.379 DEBUG:teuthology.orchestra.run.smithi145:> uname -m 2024-04-07T13:52:09.398 INFO:teuthology.orchestra.run.smithi145.stdout:x86_64 2024-04-07T13:52:09.398 DEBUG:teuthology.orchestra.run.smithi145:> cat /etc/os-release 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:NAME="CentOS Stream" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION="9" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:ID="centos" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:ID_LIKE="rhel fedora" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:VERSION_ID="9" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:PLATFORM_ID="platform:el9" 2024-04-07T13:52:09.455 INFO:teuthology.orchestra.run.smithi145.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:ANSI_COLOR="0;31" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:LOGO="fedora-logo-icon" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:HOME_URL="https://centos.org/" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-07T13:52:09.456 INFO:teuthology.orchestra.run.smithi145.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-07T13:52:09.456 INFO:teuthology.lock.ops:Updating smithi145.front.sepia.ceph.com on lock server 2024-04-07T13:52:09.479 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-07T13:52:09.488 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-07T13:52:09.493 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-07T13:52:09.493 DEBUG:teuthology.orchestra.run.smithi111:> test '!' -e /home/ubuntu/cephtest 2024-04-07T13:52:09.496 DEBUG:teuthology.orchestra.run.smithi145:> test '!' -e /home/ubuntu/cephtest 2024-04-07T13:52:09.511 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-07T13:52:09.517 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-07T13:52:09.518 DEBUG:teuthology.orchestra.run.smithi111:> test -z $(ls -A /var/lib/ceph) 2024-04-07T13:52:09.553 DEBUG:teuthology.orchestra.run.smithi145:> test -z $(ls -A /var/lib/ceph) 2024-04-07T13:52:09.642 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-07T13:52:09.700 INFO:teuthology.run_tasks:Running task kernel... 2024-04-07T13:52:09.714 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-04-07T13:52:09.714 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-04-07T13:52:09.714 DEBUG:teuthology.orchestra.run.smithi111:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-07T13:52:09.715 DEBUG:teuthology.orchestra.run.smithi145:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-07T13:52:09.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T13:52:09.731 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-04-07T13:52:09.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T13:52:09.732 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-04-07T13:52:09.788 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-361.el9.x86_64 2024-04-07T13:52:09.789 INFO:teuthology.task.kernel:Running kernel on smithi111: 5.14.0-361.el9.x86_64 2024-04-07T13:52:09.789 DEBUG:teuthology.orchestra.run.smithi111:> sudo yum install -y kernel 2024-04-07T13:52:09.790 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-361.el9.x86_64 2024-04-07T13:52:09.790 INFO:teuthology.task.kernel:Running kernel on smithi145: 5.14.0-361.el9.x86_64 2024-04-07T13:52:09.790 DEBUG:teuthology.orchestra.run.smithi145:> sudo yum install -y kernel 2024-04-07T13:52:11.935 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-04-07T13:52:12.029 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - AppStream 16 MB/s | 8.1 MB 00:00 2024-04-07T13:52:13.636 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.5 MB 00:00 2024-04-07T13:52:13.695 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS-9 - BaseOS 7.1 MB/s | 2.5 MB 00:00 2024-04-07T13:52:14.826 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-04-07T13:52:20.894 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - AppStream 3.8 MB/s | 19 MB 00:04 2024-04-07T13:52:22.993 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - BaseOS 948 kB/s | 8.0 MB 00:08 2024-04-07T13:52:24.935 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-04-07T13:52:26.378 INFO:teuthology.orchestra.run.smithi145.stdout:CentOS Stream 9 - Extras packages 54 kB/s | 16 kB 00:00 2024-04-07T13:52:27.490 INFO:teuthology.orchestra.run.smithi145.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-04-07T13:52:32.554 INFO:teuthology.orchestra.run.smithi145.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-04-07T13:52:34.692 INFO:teuthology.orchestra.run.smithi145.stderr:Modular dependency problems: 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 1: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 2: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 3: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 4: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 5: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 6: conflicting requests 2024-04-07T13:52:34.693 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 7: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 8: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 9: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 10: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 11: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 12: conflicting requests 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.694 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 13: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 14: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 15: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 16: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 17: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 18: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 19: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 20: conflicting requests 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.695 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 21: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 22: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 23: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 24: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 25: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 26: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 27: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 28: conflicting requests 2024-04-07T13:52:34.696 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 29: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 30: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 31: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 32: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 33: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 34: conflicting requests 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.697 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 35: conflicting requests 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 36: conflicting requests 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 37: conflicting requests 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 38: cannot install the best candidate for the job 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.698 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 39: conflicting requests 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 40: cannot install the best candidate for the job 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 41: cannot install the best candidate for the job 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.699 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 42: cannot install the best candidate for the job 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 43: cannot install the best candidate for the job 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 44: cannot install the best candidate for the job 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.700 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: Problem 45: cannot install the best candidate for the job 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-07T13:52:34.701 INFO:teuthology.orchestra.run.smithi145.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-07T13:52:35.086 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-07T13:52:35.086 INFO:teuthology.orchestra.run.smithi145.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-07T13:52:35.204 INFO:teuthology.orchestra.run.smithi145.stdout:Dependencies resolved. 2024-04-07T13:52:35.220 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-04-07T13:52:35.220 INFO:teuthology.orchestra.run.smithi145.stdout: Package Architecture Version Repository Size 2024-04-07T13:52:35.220 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Installing: 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Installing dependencies: 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction Summary 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:================================================================================ 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Install 4 Packages 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Total download size: 76 M 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Installed size: 123 M 2024-04-07T13:52:35.221 INFO:teuthology.orchestra.run.smithi145.stdout:Downloading Packages: 2024-04-07T13:52:35.464 INFO:teuthology.orchestra.run.smithi145.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 545 kB/s | 66 kB 00:00 2024-04-07T13:52:35.914 INFO:teuthology.orchestra.run.smithi145.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 26 MB/s | 15 MB 00:00 2024-04-07T13:52:36.547 INFO:teuthology.orchestra.run.smithi145.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-04-07T13:52:37.123 INFO:teuthology.orchestra.run.smithi145.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 17 MB/s | 27 MB 00:01 2024-04-07T13:52:37.123 INFO:teuthology.orchestra.run.smithi145.stdout:-------------------------------------------------------------------------------- 2024-04-07T13:52:37.123 INFO:teuthology.orchestra.run.smithi145.stdout:Total 40 MB/s | 76 MB 00:01 2024-04-07T13:52:37.785 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction check 2024-04-07T13:52:38.145 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction check succeeded. 2024-04-07T13:52:38.145 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction test 2024-04-07T13:52:39.149 INFO:teuthology.orchestra.run.smithi145.stdout:Transaction test succeeded. 2024-04-07T13:52:39.150 INFO:teuthology.orchestra.run.smithi145.stdout:Running transaction 2024-04-07T13:52:41.629 INFO:teuthology.orchestra.run.smithi145.stdout: Preparing : 1/1 2024-04-07T13:52:42.230 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-07T13:52:42.314 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:52:43.908 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - AppStream 975 kB/s | 19 MB 00:19 2024-04-07T13:52:45.061 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:52:45.226 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:52:48.109 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-04-07T13:52:49.752 INFO:teuthology.orchestra.run.smithi111.stdout:CentOS Stream 9 - Extras packages 31 kB/s | 16 kB 00:00 2024-04-07T13:52:49.953 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:52:50.110 INFO:teuthology.orchestra.run.smithi145.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:52:51.563 INFO:teuthology.orchestra.run.smithi111.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-07T13:52:54.856 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:52:56.553 INFO:teuthology.orchestra.run.smithi111.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-04-07T13:52:58.665 INFO:teuthology.orchestra.run.smithi111.stderr:Modular dependency problems: 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 1: conflicting requests 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module container-tools:rhel8:820240321091303:20125149.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 2: conflicting requests 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module eclipse:rhel8:820201023100746:b230ed4e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 3: conflicting requests 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module freeradius:3.0:8080020230117180605:89170a74.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 4: conflicting requests 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module gimp:2.8:8000020190628145146:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 5: conflicting requests 2024-04-07T13:52:58.666 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module go-toolset:rhel8:820240401160413:b754926a.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 6: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module httpd:2.4:820240216084734:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 7: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module idm:client:820240307184541:49cc9d1b.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 8: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module inkscape:0.92.3:8080020221205124429:3e031279.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 9: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module jmc:rhel8:8050020211110222101:6392b1f8.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 10: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module llvm-toolset:rhel8:820240205133135:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 11: conflicting requests 2024-04-07T13:52:58.667 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module log4j:2:8080020221205124743:9d367344.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 12: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module mailman:2.1:820230727085757:77fc8825.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 13: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module mariadb:10.3:8030020210419150013:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 14: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module maven:3.5:8060020211203142015:c0229ad2.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 15: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module mercurial:4.8:8000020190628020724:4148dfdf.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 16: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module mod_auth_openidc:2.3:820230721123832:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 17: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module mysql:8.0:8040020210901180257:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 18: conflicting requests 2024-04-07T13:52:58.668 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module nginx:1.14:8000020211221191913:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 19: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module nodejs:10:8040020210120182536:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 20: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl:5.26:8000020190628020724:55190bc5.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 21: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module php:7.2:8020020200507003613:2c7ca891.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 22: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module postgresql:10:820230223114800:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 23: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module python27:2.7:820240208011952:182f7c73.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 24: conflicting requests 2024-04-07T13:52:58.669 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module python36:3.6:8050020210825152031:982725ab.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 25: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module python38:3.8:820230810143931:64c1cd5a.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 26: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module python39:3.9:820240214182535:17377f89.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 27: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module redis:5:8040020211019153849:9f9e2e7e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 28: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module rhn-tools:1.0:8010020191114034948:f69d1239.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 29: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module ruby:2.5:820230627084142:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 30: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module rust-toolset:rhel8:820240119204620:b09eea91.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 31: conflicting requests 2024-04-07T13:52:58.670 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module satellite-5-client:1.0:8010020191114035551:cdc1202b.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 32: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module squid:4:820240319173245:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 33: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module subversion:1.10:8070020220711155714:78111232.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 34: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module swig:3.0:8030020200706142531:30b713e6.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 35: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module varnish:6:8050020211109225449:b4937e53.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 36: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module virt:rhel:820240314161907:9edba152.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 37: conflicting requests 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:7cc0a66d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:8030020200716155257:b967a9a2.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.671 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-libwww-perl:6.34:820230811133638:36f9fae6.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 38: cannot install the best candidate for the job 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:a5949e2e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:8010020191114031501:be2e4737.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:2ab761e1.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-YAML:1.24:820230223114759:86e45846.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 39: conflicting requests 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:8030020200715230104:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:082fdf2f.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-IO-Socket-SSL:2.066:820230223114759:fbe42456.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 40: cannot install the best candidate for the job 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:16b3ab4d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.672 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:8010020191114031513:e1f37755.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-FCGI:0.78:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 41: cannot install the best candidate for the job 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:1e4bbb35.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:8030020200716150652:ea09926d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:0ccef39c.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBI:1.641:820230223114759:dbc5bc9a.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 42: cannot install the best candidate for the job 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:8030020200716174729:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-SQLite:1.58:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.673 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 43: cannot install the best candidate for the job 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:56fce90f.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:8030020200716213634:98a3c9d0.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:06b7596e.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-Pg:3.7:820230223114759:d2fc5d76.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 44: cannot install the best candidate for the job 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:3a70019f.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:8030020200716171139:cccafca5.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.674 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:6fdf7e5d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-DBD-MySQL:4.046:820230223114759:d4326aba.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: Problem 45: cannot install the best candidate for the job 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8010020191120175858:95fce66d.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:8030020200723233049:09acf126.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:436af1ae.x86_64 from CentOS-AppStream 2024-04-07T13:52:58.677 INFO:teuthology.orchestra.run.smithi111.stderr: - nothing provides module(platform:el8) needed by module perl-App-cpanminus:1.7044:820230223114759:b12195ea.x86_64 from CentOS-AppStream 2024-04-07T13:52:59.035 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-04-07T13:52:59.035 INFO:teuthology.orchestra.run.smithi111.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-04-07T13:52:59.140 INFO:teuthology.orchestra.run.smithi111.stdout:Dependencies resolved. 2024-04-07T13:52:59.151 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: Package Architecture Version Repository Size 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:Installing: 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: kernel x86_64 5.14.0-432.el9 baseos 66 k 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:Installing dependencies: 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core x86_64 5.14.0-432.el9 baseos 15 M 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules x86_64 5.14.0-432.el9 baseos 33 M 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core x86_64 5.14.0-432.el9 baseos 27 M 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction Summary 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:================================================================================ 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout:Install 4 Packages 2024-04-07T13:52:59.152 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:52:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:Total download size: 76 M 2024-04-07T13:52:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:Installed size: 123 M 2024-04-07T13:52:59.153 INFO:teuthology.orchestra.run.smithi111.stdout:Downloading Packages: 2024-04-07T13:52:59.312 INFO:teuthology.orchestra.run.smithi111.stdout:(1/4): kernel-5.14.0-432.el9.x86_64.rpm 538 kB/s | 66 kB 00:00 2024-04-07T13:52:59.838 INFO:teuthology.orchestra.run.smithi111.stdout:(2/4): kernel-core-5.14.0-432.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-04-07T13:53:00.379 INFO:teuthology.orchestra.run.smithi111.stdout:(3/4): kernel-modules-5.14.0-432.el9.x86_64.rpm 28 MB/s | 33 MB 00:01 2024-04-07T13:53:00.797 INFO:teuthology.orchestra.run.smithi111.stdout:(4/4): kernel-modules-core-5.14.0-432.el9.x86_6 19 MB/s | 27 MB 00:01 2024-04-07T13:53:00.797 INFO:teuthology.orchestra.run.smithi111.stdout:-------------------------------------------------------------------------------- 2024-04-07T13:53:00.797 INFO:teuthology.orchestra.run.smithi111.stdout:Total 46 MB/s | 76 MB 00:01 2024-04-07T13:53:01.460 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction check 2024-04-07T13:53:01.841 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction check succeeded. 2024-04-07T13:53:01.841 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction test 2024-04-07T13:53:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:Transaction test succeeded. 2024-04-07T13:53:02.827 INFO:teuthology.orchestra.run.smithi111.stdout:Running transaction 2024-04-07T13:53:05.271 INFO:teuthology.orchestra.run.smithi111.stdout: Preparing : 1/1 2024-04-07T13:53:05.895 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-core-5.14.0-432.el9.x86_64 1/4 2024-04-07T13:53:05.988 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:53:08.756 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:53:08.907 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:53:13.650 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:53:13.757 INFO:teuthology.orchestra.run.smithi111.stdout: Installing : kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:18.525 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:24.050 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:24.050 INFO:teuthology.orchestra.run.smithi145.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-07T13:53:24.050 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:53:24.094 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:24.519 INFO:teuthology.orchestra.run.smithi145.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:24.519 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-07T13:53:24.519 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:53:24.519 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout:Installed: 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-07T13:53:24.976 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:53:24.977 INFO:teuthology.orchestra.run.smithi145.stdout:Complete! 2024-04-07T13:53:25.325 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T13:53:25.425 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:25.425 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T13:53:25.425 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-07T13:53:25.425 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-07T13:53:25.425 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-07T13:53:25.426 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-04-07T13:53:25.426 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi145.front.sepia.ceph.com, path=None, version=distro) 2024-04-07T13:53:25.426 DEBUG:teuthology.orchestra.run.smithi145:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T13:53:25.505 INFO:teuthology.orchestra.run.smithi145.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:25.505 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T13:53:25.505 DEBUG:teuthology.orchestra.run.smithi145:> sudo rpm -qi grub2-tools 2024-04-07T13:53:25.539 INFO:teuthology.orchestra.run.smithi145.stdout:Name : grub2-tools 2024-04-07T13:53:25.539 INFO:teuthology.orchestra.run.smithi145.stdout:Epoch : 1 2024-04-07T13:53:25.539 INFO:teuthology.orchestra.run.smithi145.stdout:Version : 2.06 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Release : 68.el9 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Architecture: x86_64 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Group : Unspecified 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Size : 8264254 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:License : GPLv3+ 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Packager : builder@centos.org 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Vendor : CentOS 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-07T13:53:25.540 INFO:teuthology.orchestra.run.smithi145.stdout:Summary : Support tools for GRUB. 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:Description : 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:hardware devices. 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T13:53:25.541 INFO:teuthology.orchestra.run.smithi145.stdout:This subpackage provides tools for support of all platforms. 2024-04-07T13:53:25.542 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-07T13:53:25.542 INFO:teuthology.task.kernel:Updating grub on smithi145 to boot 5.14.0-432.el9.x86_64 2024-04-07T13:53:25.542 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-07T13:53:26.159 INFO:teuthology.orchestra.run.smithi145.stderr:Generating grub configuration file ... 2024-04-07T13:53:27.522 INFO:teuthology.orchestra.run.smithi145.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-07T13:53:27.569 INFO:teuthology.orchestra.run.smithi145.stderr:done 2024-04-07T13:53:27.572 DEBUG:teuthology.orchestra.run.smithi145:> mktemp 2024-04-07T13:53:27.587 INFO:teuthology.orchestra.run.smithi145.stdout:/tmp/tmp.pQYrzHmbQl 2024-04-07T13:53:27.587 DEBUG:teuthology.orchestra.run.smithi145:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.pQYrzHmbQl 2024-04-07T13:53:27.656 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 0666 /tmp/tmp.pQYrzHmbQl 2024-04-07T13:53:27.795 DEBUG:teuthology.orchestra.remote:smithi145:/tmp/tmp.pQYrzHmbQl is 6KB 2024-04-07T13:53:27.814 DEBUG:teuthology.orchestra.run.smithi145:> rm -fr /tmp/tmp.pQYrzHmbQl 2024-04-07T13:53:27.828 DEBUG:teuthology.orchestra.run.smithi145:> sudo /bin/ls /boot/loader/entries || true 2024-04-07T13:53:27.893 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-07T13:53:27.893 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-07T13:53:27.893 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-07T13:53:27.893 INFO:teuthology.orchestra.run.smithi145.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-07T13:53:27.894 DEBUG:teuthology.orchestra.run.smithi145:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-07T13:53:28.057 DEBUG:teuthology.orchestra.run.smithi145:> sudo shutdown -r now 2024-04-07T13:53:47.747 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:47.747 INFO:teuthology.orchestra.run.smithi111.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-04-07T13:53:47.747 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:53:47.791 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-modules-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:48.141 INFO:teuthology.orchestra.run.smithi111.stdout: Running scriptlet: kernel-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:48.141 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-5.14.0-432.el9.x86_64 1/4 2024-04-07T13:53:48.141 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-core-5.14.0-432.el9.x86_64 2/4 2024-04-07T13:53:48.141 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-5.14.0-432.el9.x86_64 3/4 2024-04-07T13:53:48.598 INFO:teuthology.orchestra.run.smithi111.stdout: Verifying : kernel-modules-core-5.14.0-432.el9.x86_64 4/4 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout:Installed: 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-core-5.14.0-432.el9.x86_64 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-5.14.0-432.el9.x86_64 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: kernel-modules-core-5.14.0-432.el9.x86_64 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:53:48.599 INFO:teuthology.orchestra.run.smithi111.stdout:Complete! 2024-04-07T13:53:48.930 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T13:53:49.051 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:49.051 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T13:53:49.051 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-432.el9.x86_64 2024-04-07T13:53:49.051 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-04-07T13:53:49.051 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-04-07T13:53:49.052 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-04-07T13:53:49.052 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi111.front.sepia.ceph.com, path=None, version=distro) 2024-04-07T13:53:49.052 DEBUG:teuthology.orchestra.run.smithi111:> rpm -q kernel | sort -rV | head -n 1 2024-04-07T13:53:49.132 INFO:teuthology.orchestra.run.smithi111.stdout:kernel-5.14.0-432.el9.x86_64 2024-04-07T13:53:49.132 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-432.el9.x86_64 2024-04-07T13:53:49.132 DEBUG:teuthology.orchestra.run.smithi111:> sudo rpm -qi grub2-tools 2024-04-07T13:53:49.166 INFO:teuthology.orchestra.run.smithi111.stdout:Name : grub2-tools 2024-04-07T13:53:49.166 INFO:teuthology.orchestra.run.smithi111.stdout:Epoch : 1 2024-04-07T13:53:49.166 INFO:teuthology.orchestra.run.smithi111.stdout:Version : 2.06 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Release : 68.el9 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Architecture: x86_64 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Group : Unspecified 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Size : 8264254 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:License : GPLv3+ 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Packager : builder@centos.org 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Vendor : CentOS 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Summary : Support tools for GRUB. 2024-04-07T13:53:49.167 INFO:teuthology.orchestra.run.smithi111.stdout:Description : 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout:hardware devices. 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T13:53:49.168 INFO:teuthology.orchestra.run.smithi111.stdout:This subpackage provides tools for support of all platforms. 2024-04-07T13:53:49.169 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-07T13:53:49.169 INFO:teuthology.task.kernel:Updating grub on smithi111 to boot 5.14.0-432.el9.x86_64 2024-04-07T13:53:49.169 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-07T13:53:49.790 INFO:teuthology.orchestra.run.smithi111.stderr:Generating grub configuration file ... 2024-04-07T13:53:51.114 INFO:teuthology.orchestra.run.smithi111.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-07T13:53:51.152 INFO:teuthology.orchestra.run.smithi111.stderr:done 2024-04-07T13:53:51.154 DEBUG:teuthology.orchestra.run.smithi111:> mktemp 2024-04-07T13:53:51.169 INFO:teuthology.orchestra.run.smithi111.stdout:/tmp/tmp.dkpV9tUjXa 2024-04-07T13:53:51.170 DEBUG:teuthology.orchestra.run.smithi111:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.dkpV9tUjXa 2024-04-07T13:53:51.237 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0666 /tmp/tmp.dkpV9tUjXa 2024-04-07T13:53:51.372 DEBUG:teuthology.orchestra.remote:smithi111:/tmp/tmp.dkpV9tUjXa is 6KB 2024-04-07T13:53:51.421 DEBUG:teuthology.orchestra.run.smithi111:> rm -fr /tmp/tmp.dkpV9tUjXa 2024-04-07T13:53:51.435 DEBUG:teuthology.orchestra.run.smithi111:> sudo /bin/ls /boot/loader/entries || true 2024-04-07T13:53:51.502 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-04-07T13:53:51.502 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-04-07T13:53:51.502 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-04-07T13:53:51.502 INFO:teuthology.orchestra.run.smithi111.stdout:15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64.conf 2024-04-07T13:53:51.503 DEBUG:teuthology.orchestra.run.smithi111:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-432.el9.x86_64 2024-04-07T13:53:51.640 DEBUG:teuthology.orchestra.run.smithi111:> sudo shutdown -r now 2024-04-07T13:53:58.066 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-04-07T13:53:58.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T13:53:58.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:54:16.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-04-07T13:54:21.645 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-04-07T13:54:21.645 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:54:21.646 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:54:25.482 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T13:54:25.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:54:28.542 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.145 2024-04-07T13:54:40.030 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-04-07T13:54:40.543 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T13:54:40.544 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:54:49.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:54:49.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:54:52.094 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-04-07T13:55:04.107 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:55:04.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:55:07.166 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-04-07T13:55:12.344 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-04-07T13:55:12.795 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T13:55:12.795 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-432.el9.x86_64"... 2024-04-07T13:55:12.796 DEBUG:teuthology.orchestra.run.smithi145:> uname -r 2024-04-07T13:55:12.852 INFO:teuthology.orchestra.run.smithi145.stdout:5.14.0-432.el9.x86_64 2024-04-07T13:55:12.853 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-07T13:55:12.853 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-07T13:55:12.853 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-07T13:55:13.853 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-04-07T13:55:13.854 DEBUG:teuthology.orchestra.run.smithi145:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-07T13:55:13.915 INFO:teuthology.orchestra.run.smithi145.stdout:ttyS1 2024-04-07T13:55:13.955 DEBUG:teuthology.parallel:result is None 2024-04-07T13:55:22.167 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:55:22.168 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:55:40.706 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.111 2024-04-07T13:55:58.708 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:55:58.709 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T13:55:59.307 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-07T13:55:59.711 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T13:55:59.711 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-432.el9.x86_64"... 2024-04-07T13:55:59.711 DEBUG:teuthology.orchestra.run.smithi111:> uname -r 2024-04-07T13:55:59.728 INFO:teuthology.orchestra.run.smithi111.stdout:5.14.0-432.el9.x86_64 2024-04-07T13:55:59.728 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-432.el9.x86_64 vs 5.14.0-432.el9.x86_64 2024-04-07T13:55:59.729 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-07T13:55:59.729 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-07T13:56:00.730 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-04-07T13:56:00.731 DEBUG:teuthology.orchestra.run.smithi111:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-04-07T13:56:00.809 INFO:teuthology.orchestra.run.smithi111.stdout:ttyS1 2024-04-07T13:56:00.848 DEBUG:teuthology.parallel:result is None 2024-04-07T13:56:00.848 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-07T13:56:00.856 INFO:teuthology.task.internal:Creating test directory... 2024-04-07T13:56:00.856 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-07T13:56:00.859 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-07T13:56:00.877 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-07T13:56:00.885 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-07T13:56:00.905 INFO:teuthology.task.internal:Creating archive directory... 2024-04-07T13:56:00.905 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-07T13:56:00.916 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-07T13:56:00.971 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-07T13:56:00.979 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-07T13:56:00.979 DEBUG:teuthology.orchestra.run.smithi111:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-07T13:56:00.994 DEBUG:teuthology.orchestra.run.smithi145:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-07T13:56:01.024 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T13:56:01.037 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T13:56:01.041 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T13:56:01.055 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-07T13:56:01.057 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-07T13:56:01.064 INFO:teuthology.task.internal:Configuring sudo... 2024-04-07T13:56:01.065 DEBUG:teuthology.orchestra.run.smithi111:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-07T13:56:01.081 DEBUG:teuthology.orchestra.run.smithi145:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-07T13:56:01.110 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-07T13:56:01.121 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-07T13:56:01.121 DEBUG:teuthology.orchestra.run.smithi111:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-07T13:56:01.149 DEBUG:teuthology.orchestra.run.smithi145:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-07T13:56:01.167 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T13:56:01.241 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T13:56:01.292 DEBUG:teuthology.orchestra.run.smithi111:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T13:56:01.367 DEBUG:teuthology.orchestra.run.smithi111:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T13:56:01.415 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T13:56:01.415 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-07T13:56:01.483 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T13:56:01.523 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-07T13:56:01.584 DEBUG:teuthology.orchestra.run.smithi145:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T13:56:01.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-07T13:56:01.713 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T13:56:01.713 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-07T13:56:01.783 DEBUG:teuthology.orchestra.run.smithi111:> sudo service rsyslog restart 2024-04-07T13:56:01.785 DEBUG:teuthology.orchestra.run.smithi145:> sudo service rsyslog restart 2024-04-07T13:56:01.830 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T13:56:01.878 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T13:56:02.309 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-07T13:56:02.317 INFO:teuthology.task.internal:Starting timer... 2024-04-07T13:56:02.317 INFO:teuthology.run_tasks:Running task pcp... 2024-04-07T13:56:02.334 INFO:teuthology.run_tasks:Running task selinux... 2024-04-07T13:56:02.349 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-07T13:56:02.349 DEBUG:teuthology.orchestra.run.smithi111:> sudo service auditd rotate 2024-04-07T13:56:02.393 INFO:teuthology.orchestra.run.smithi111.stdout:Rotating logs: 2024-04-07T13:56:02.394 DEBUG:teuthology.orchestra.run.smithi145:> sudo service auditd rotate 2024-04-07T13:56:02.458 INFO:teuthology.orchestra.run.smithi145.stdout:Rotating logs: 2024-04-07T13:56:02.460 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-07T13:56:02.460 DEBUG:teuthology.orchestra.run.smithi111:> /usr/sbin/getenforce 2024-04-07T13:56:02.490 INFO:teuthology.orchestra.run.smithi111.stdout:Permissive 2024-04-07T13:56:02.490 DEBUG:teuthology.orchestra.run.smithi145:> /usr/sbin/getenforce 2024-04-07T13:56:02.542 INFO:teuthology.orchestra.run.smithi145.stdout:Permissive 2024-04-07T13:56:02.543 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi111.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi145.front.sepia.ceph.com': 'permissive'} 2024-04-07T13:56:02.543 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-07T13:56:02.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T13:56:02.571 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-07T13:56:02.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T13:56:02.600 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-07T13:56:02.601 DEBUG:teuthology.orchestra.run.smithi111:> sudo /usr/sbin/setenforce permissive 2024-04-07T13:56:02.642 DEBUG:teuthology.orchestra.run.smithi145:> sudo /usr/sbin/setenforce permissive 2024-04-07T13:56:02.678 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-07T13:56:02.688 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-07T13:56:02.706 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-07T13:56:02.708 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi111.front.sepia.ceph.com,smithi145.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-07T14:00:20.138 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi111.front.sepia.ceph.com'), Remote(name='ubuntu@smithi145.front.sepia.ceph.com')] 2024-04-07T14:00:20.139 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T14:00:20.140 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi111.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T14:00:20.229 DEBUG:teuthology.orchestra.run.smithi111:> true 2024-04-07T14:00:20.262 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi111.front.sepia.ceph.com' 2024-04-07T14:00:20.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T14:00:20.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi145.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-07T14:00:20.350 DEBUG:teuthology.orchestra.run.smithi145:> true 2024-04-07T14:00:20.433 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi145.front.sepia.ceph.com' 2024-04-07T14:00:20.433 INFO:teuthology.run_tasks:Running task clock... 2024-04-07T14:00:20.443 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-07T14:00:20.444 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-07T14:00:20.444 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T14:00:20.446 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-07T14:00:20.446 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T14:00:20.476 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-07T14:00:20.492 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-07T14:00:20.504 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-07T14:00:20.520 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-07T14:00:20.521 INFO:teuthology.orchestra.run.smithi111.stderr:sudo: ntpd: command not found 2024-04-07T14:00:20.535 INFO:teuthology.orchestra.run.smithi111.stdout:506 Cannot talk to daemon 2024-04-07T14:00:20.549 INFO:teuthology.orchestra.run.smithi145.stderr:sudo: ntpd: command not found 2024-04-07T14:00:20.551 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-07T14:00:20.563 INFO:teuthology.orchestra.run.smithi145.stdout:506 Cannot talk to daemon 2024-04-07T14:00:20.567 INFO:teuthology.orchestra.run.smithi111.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-07T14:00:20.579 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-07T14:00:20.596 INFO:teuthology.orchestra.run.smithi145.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-07T14:00:20.622 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-07T14:00:20.626 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T14:00:20.626 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-07T14:00:20.627 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.627 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.627 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.627 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.648 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.651 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:00:20.652 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-07T14:00:20.748 INFO:tasks.cephadm:Config: {'cephadm_branch': 'v17.2.0', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'quay.io/ceph/ceph:v17.2.0', 'roleless': True, 'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'e5c885fe95ee982ff7b45214daa61b1a6d09d57d'} 2024-04-07T14:00:20.748 INFO:tasks.cephadm:Cluster image is quay.io/ceph/ceph:v17.2.0 2024-04-07T14:00:20.749 INFO:tasks.cephadm:Cluster fsid is 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:20.749 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-07T14:00:20.749 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-04-07T14:00:20.749 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi111': '172.21.15.111', 'mon.smithi145': '172.21.15.145'} 2024-04-07T14:00:20.749 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-07T14:00:20.750 DEBUG:teuthology.orchestra.run.smithi111:> sudo hostname $(hostname -s) 2024-04-07T14:00:20.777 DEBUG:teuthology.orchestra.run.smithi145:> sudo hostname $(hostname -s) 2024-04-07T14:00:20.805 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v17.2.0)... 2024-04-07T14:00:20.805 DEBUG:teuthology.orchestra.run.smithi111:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.067 INFO:teuthology.orchestra.run.smithi111.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 7 14:00 /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.067 DEBUG:teuthology.orchestra.run.smithi145:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v17.2.0/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.230 INFO:teuthology.orchestra.run.smithi145.stdout:-rw-r--r--. 1 ubuntu ubuntu 320521 Apr 7 14:00 /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.230 DEBUG:teuthology.orchestra.run.smithi111:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.246 DEBUG:teuthology.orchestra.run.smithi145:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-07T14:00:21.269 INFO:tasks.cephadm:Pulling image quay.io/ceph/ceph:v17.2.0 on all hosts... 2024-04-07T14:00:21.269 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-07T14:00:21.289 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 pull 2024-04-07T14:00:22.167 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T14:00:22.171 INFO:teuthology.orchestra.run.smithi145.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T14:00:39.617 INFO:teuthology.orchestra.run.smithi145.stdout:{ 2024-04-07T14:00:39.617 INFO:teuthology.orchestra.run.smithi145.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-07T14:00:39.617 INFO:teuthology.orchestra.run.smithi145.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-07T14:00:39.617 INFO:teuthology.orchestra.run.smithi145.stdout: "repo_digests": [ 2024-04-07T14:00:39.617 INFO:teuthology.orchestra.run.smithi145.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-07T14:00:39.618 INFO:teuthology.orchestra.run.smithi145.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-07T14:00:39.618 INFO:teuthology.orchestra.run.smithi145.stdout: ] 2024-04-07T14:00:39.618 INFO:teuthology.orchestra.run.smithi145.stdout:} 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph_version": "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)", 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: "image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: "repo_digests": [ 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: "quay.io/ceph/ceph@sha256:cb4d698cb769b6aba05bf6ef04f41a7fe694160140347576e13bd9348514b667" 2024-04-07T14:00:41.307 INFO:teuthology.orchestra.run.smithi111.stdout: ] 2024-04-07T14:00:41.308 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:00:41.326 DEBUG:teuthology.orchestra.run.smithi111:> sudo mkdir -p /etc/ceph 2024-04-07T14:00:41.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo mkdir -p /etc/ceph 2024-04-07T14:00:41.406 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 777 /etc/ceph 2024-04-07T14:00:41.438 DEBUG:teuthology.orchestra.run.smithi145:> sudo chmod 777 /etc/ceph 2024-04-07T14:00:41.476 INFO:tasks.cephadm:Writing seed config... 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-07T14:00:41.477 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-07T14:00:41.478 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:00:41.478 DEBUG:teuthology.orchestra.run.smithi111:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-07T14:00:41.493 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-04-07T14:00:41.494 DEBUG:teuthology.orchestra.run.smithi111:mon.smithi111> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service 2024-04-07T14:00:41.536 INFO:tasks.cephadm:Bootstrapping... 2024-04-07T14:00:41.536 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 -v bootstrap --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.111 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:00:41.730 INFO:teuthology.orchestra.run.smithi111.stderr:-------------------------------------------------------------------------------- 2024-04-07T14:00:41.730 INFO:teuthology.orchestra.run.smithi111.stderr:cephadm ['--image', 'quay.io/ceph/ceph:v17.2.0', '-v', 'bootstrap', '--fsid', '2bfa0756-f4e7-11ee-b649-cb9ed24678a4', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.111', '--skip-admin-label'] 2024-04-07T14:00:41.753 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.4-dev 2024-04-07T14:00:41.755 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying podman|docker is present... 2024-04-07T14:00:41.776 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.4-dev 2024-04-07T14:00:41.778 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying lvm2 is present... 2024-04-07T14:00:41.778 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying time synchronization is in place... 2024-04-07T14:00:41.785 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-07T14:00:41.791 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:00:41.797 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: enabled 2024-04-07T14:00:41.803 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: active 2024-04-07T14:00:41.804 INFO:teuthology.orchestra.run.smithi111.stderr:Unit chronyd.service is enabled and running 2024-04-07T14:00:41.804 INFO:teuthology.orchestra.run.smithi111.stderr:Repeating the final host check... 2024-04-07T14:00:41.824 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: 4.9.4-dev 2024-04-07T14:00:41.826 INFO:teuthology.orchestra.run.smithi111.stderr:podman (/bin/podman) version 4.9.4 is present 2024-04-07T14:00:41.827 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl is present 2024-04-07T14:00:41.827 INFO:teuthology.orchestra.run.smithi111.stderr:lvcreate is present 2024-04-07T14:00:41.832 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-04-07T14:00:41.838 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:00:41.844 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: enabled 2024-04-07T14:00:41.850 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: active 2024-04-07T14:00:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:Unit chronyd.service is enabled and running 2024-04-07T14:00:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:Host looks OK 2024-04-07T14:00:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:Cluster fsid: 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:Acquiring lock 140466305080720 on /run/cephadm/2bfa0756-f4e7-11ee-b649-cb9ed24678a4.lock 2024-04-07T14:00:41.851 INFO:teuthology.orchestra.run.smithi111.stderr:Lock 140466305080720 acquired on /run/cephadm/2bfa0756-f4e7-11ee-b649-cb9ed24678a4.lock 2024-04-07T14:00:41.852 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying IP 172.21.15.111 port 3300 ... 2024-04-07T14:00:41.852 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying IP 172.21.15.111 port 6789 ... 2024-04-07T14:00:41.852 INFO:teuthology.orchestra.run.smithi111.stderr:Base mon IP is 172.21.15.111, final addrv is [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-04-07T14:00:41.854 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.111 metric 100 2024-04-07T14:00:41.854 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.111 metric 100 2024-04-07T14:00:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-04-07T14:00:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-04-07T14:00:41.857 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2024-04-07T14:00:41.859 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-07T14:00:41.859 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-04-07T14:00:41.859 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-07T14:00:41.859 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-04-07T14:00:41.859 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cc2f/64 scope link 2024-04-07T14:00:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-04-07T14:00:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:Mon IP `172.21.15.111` is in CIDR network `172.21.0.0/20` 2024-04-07T14:00:41.860 INFO:teuthology.orchestra.run.smithi111.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-07T14:00:41.861 INFO:teuthology.orchestra.run.smithi111.stderr:Pulling container image quay.io/ceph/ceph:v17.2.0... 2024-04-07T14:00:41.963 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Trying to pull quay.io/ceph/ceph:v17.2.0... 2024-04-07T14:00:42.337 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Getting image source signatures 2024-04-07T14:00:42.337 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:33ca8fff7868c4dc0c11e09bca97c720eb9cfbab7221216754367dd8de70388a 2024-04-07T14:00:42.337 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:c32ab78b488d0c72f64eded765c0cf6b5bf2c75dab66cb62a9d367fa6ec42513 2024-04-07T14:00:42.337 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:a70843738bb77e1ab9c1f85969ebdfa55f178e746be081d1cb4f94011f69eb7c 2024-04-07T14:00:42.338 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:599d07cb321ff0a3c82224e1138fc685793fa69b93ed5780415751a5f7e4b8c2 2024-04-07T14:00:42.338 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying blob sha256:89b4a75bc2d8500f15463747507c9623df43886c134463e7f0527e70900e7a7b 2024-04-07T14:00:42.338 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Copying config sha256:e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-07T14:00:42.494 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: Writing manifest to image destination 2024-04-07T14:00:42.638 INFO:teuthology.orchestra.run.smithi111.stderr:/bin/podman: e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 2024-04-07T14:00:43.046 INFO:teuthology.orchestra.run.smithi111.stderr:ceph: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-07T14:00:43.604 INFO:teuthology.orchestra.run.smithi111.stderr:Ceph version: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable) 2024-04-07T14:00:43.605 INFO:teuthology.orchestra.run.smithi111.stderr:Extracting ceph user uid/gid from container image... 2024-04-07T14:00:43.996 INFO:teuthology.orchestra.run.smithi111.stderr:stat: 167 167 2024-04-07T14:00:44.551 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial keys... 2024-04-07T14:00:44.984 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQAMpxJmJd6sNRAA8J8vvoHo6G71IRD8EYHtDA== 2024-04-07T14:00:45.873 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQANpxJmtAyrLhAAesuyw6qwIDThMefC4zsuRA== 2024-04-07T14:00:46.970 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph-authtool: AQAOpxJmf7xANRAAQzz1T7tzJOaf4U+pC+IEeA== 2024-04-07T14:00:47.493 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial monmap... 2024-04-07T14:00:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-07T14:00:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2024-04-07T14:00:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:47.910 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-07T14:00:48.391 INFO:teuthology.orchestra.run.smithi111.stderr:monmaptool for smithi111 [v2:172.21.15.111:3300,v1:172.21.15.111:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-07T14:00:48.391 INFO:teuthology.orchestra.run.smithi111.stderr:setting min_mon_release = octopus 2024-04-07T14:00:48.391 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: set fsid to 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:48.391 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-07T14:00:48.391 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:00:48.392 INFO:teuthology.orchestra.run.smithi111.stderr:Creating mon... 2024-04-07T14:00:49.347 INFO:teuthology.orchestra.run.smithi111.stderr:create mon.smithi111 on 2024-04-07T14:00:49.554 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-07T14:00:49.703 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target. 2024-04-07T14:00:49.703 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target → /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target. 2024-04-07T14:00:49.999 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to reset failed state of unit ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service: Unit ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service not loaded. 2024-04-07T14:00:50.006 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target.wants/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service → /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@.service. 2024-04-07T14:00:50.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 podman[24439]: 2024-04-07 14:00:50.310519345 +0000 UTC m=+0.018484647 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-07T14:00:50.709 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-04-07T14:00:50.715 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:00:50.716 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-04-07T14:00:50.716 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to enable service . firewalld.service is not available 2024-04-07T14:00:50.717 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mon to start... 2024-04-07T14:00:50.717 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mon... 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 podman[24439]: 2024-04-07 14:00:50.598456706 +0000 UTC m=+0.306421981 container init f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, io.openshift.expose-services=, version=8, io.openshift.tags=base centos centos-stream, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, com.redhat.component=centos-stream-container, ceph=True, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, name=centos-stream, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 podman[24439]: 2024-04-07 14:00:50.601294806 +0000 UTC m=+0.309260083 container start f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , architecture=x86_64, distribution-scope=public) 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 ceph-mon[24458]: mkfs 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 ceph-mon[24458]: mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 bash[24439]: f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd 2024-04-07T14:00:50.841 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:50 smithi111 systemd[1]: Started Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4. 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: cluster: 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: id: 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: services: 2024-04-07T14:00:51.214 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi111 (age 0.55387s) 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mgr: no daemons active 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: data: 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: pgs: 2024-04-07T14:00:51.215 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: fsmap 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: osdmap e1: 0 total, 0 up, 0 in 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: mgrmap e1: no daemons active 2024-04-07T14:00:51.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:51 smithi111 ceph-mon[24458]: from='client.? 172.21.15.111:0/3264175768' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-07T14:00:51.722 INFO:teuthology.orchestra.run.smithi111.stderr:mon is available 2024-04-07T14:00:51.722 INFO:teuthology.orchestra.run.smithi111.stderr:Assimilating anything we can from ceph.conf... 2024-04-07T14:00:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [global] 2024-04-07T14:00:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: fsid = 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.111:3300,v1:172.21.15.111:6789] 2024-04-07T14:00:52.412 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [mgr] 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [osd] 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-07T14:00:52.413 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-07T14:00:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:52 smithi111 ceph-mon[24458]: from='client.? 172.21.15.111:0/700539198' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-07T14:00:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:52 smithi111 ceph-mon[24458]: from='client.? 172.21.15.111:0/700539198' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-07T14:00:53.019 INFO:teuthology.orchestra.run.smithi111.stderr:Generating new minimal ceph.conf... 2024-04-07T14:00:53.909 INFO:teuthology.orchestra.run.smithi111.stderr:Restarting the monitor... 2024-04-07T14:00:53.924 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:53 smithi111 ceph-mon[24458]: from='client.? 172.21.15.111:0/1872683003' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:00:53.924 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:53 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:00:54.222 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:53 smithi111 bash[24677]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon.smithi111" found: no such container 2024-04-07T14:00:54.222 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111[24454]: 2024-04-07T14:00:54.069+0000 7f94b6a9d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:00:54.222 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111[24454]: 2024-04-07T14:00:54.069+0000 7f94b6a9d700 -1 mon.smithi111@0(leader) e1 *** Got Signal Terminated *** 2024-04-07T14:00:54.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 podman[24688]: 2024-04-07 14:00:54.223366263 +0000 UTC m=+0.264157507 container died f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, maintainer=Guillaume Abrioux , name=centos-stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.openshift.expose-services=, distribution-scope=public, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git) 2024-04-07T14:00:54.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 podman[24688]: 2024-04-07 14:00:54.576997963 +0000 UTC m=+0.617789218 container cleanup f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, GIT_CLEAN=True, version=8, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, release=754, ceph=True, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , RELEASE=HEAD) 2024-04-07T14:00:54.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 bash[24688]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111 2024-04-07T14:00:54.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 bash[24715]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon.smithi111" found: no such container 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 podman[24702]: 2024-04-07 14:00:54.836201999 +0000 UTC m=+0.608471123 container remove f19c63837a74468de1d5333cb6cf8ae056848d0fab4d6836afea7da8fb7b29cd (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, vendor=Red Hat, Inc., distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, io.openshift.expose-services=, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 systemd[1]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service: Deactivated successfully. 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 systemd[1]: Stopped Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4. 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:54 smithi111 systemd[1]: Starting Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 podman[24784]: 2024-04-07 14:00:55.045638475 +0000 UTC m=+0.097003570 container create 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, name=centos-stream, vcs-type=git, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , ceph=True, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.openshift.expose-services=, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T14:00:55.116 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 podman[24784]: 2024-04-07 14:00:54.967319308 +0000 UTC m=+0.018684403 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph:v17.2.0 2024-04-07T14:00:55.257 INFO:teuthology.orchestra.run.smithi111.stderr:Setting mon public_network to 172.21.0.0/20 2024-04-07T14:00:55.380 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 podman[24784]: 2024-04-07 14:00:55.189742269 +0000 UTC m=+0.241107379 container init 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., RELEASE=HEAD, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , release=754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True) 2024-04-07T14:00:55.380 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 podman[24784]: 2024-04-07 14:00:55.192937479 +0000 UTC m=+0.244302576 container start 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, release=754, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, GIT_CLEAN=True) 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: set uid:gid to 167:167 (ceph:ceph) 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: pidfile_write: ignore empty --pid-file 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: load: jerasure load: lrc 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: RocksDB version: 6.15.5 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-07T14:00:55.381 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Compile date Apr 18 2022 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: DB SUMMARY 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: DB Session ID: POPIE55MGO7T4SRVE0YQ 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: CURRENT file: CURRENT 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: IDENTITY file: IDENTITY 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi111/store.db dir, Total Num: 1, files: 000008.sst 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi111/store.db: 000010.log size: 82735 ; 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.error_if_exists: 0 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.create_if_missing: 0 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.paranoid_checks: 1 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.env: 0x55ad956c6860 2024-04-07T14:00:55.382 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.fs: Posix File System 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.info_log: 0x55ad977ddfa0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_file_opening_threads: 16 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.statistics: (nil) 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.use_fsync: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_log_file_size: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.keep_log_file_num: 1000 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.recycle_log_file_num: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_fallocate: 1 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_mmap_reads: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_mmap_writes: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.use_direct_reads: 0 2024-04-07T14:00:55.383 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.create_missing_column_families: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.db_log_dir: 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi111/store.db 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.advise_random_on_open: 1 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.db_write_buffer_size: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.write_buffer_manager: 0x55ad978ce2d0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-07T14:00:55.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.rate_limiter: (nil) 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.wal_recovery_mode: 2 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.enable_thread_tracking: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.enable_pipelined_write: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.unordered_write: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.row_cache: None 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.wal_filter: None 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-07T14:00:55.385 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_ingest_behind: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.preserve_deletes: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.two_write_queues: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.manual_wal_flush: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.atomic_flush: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.log_readahead_size: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.best_efforts_recovery: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.allow_data_in_errors: 0 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.db_host_id: __hostname__ 2024-04-07T14:00:55.386 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_background_jobs: 2 2024-04-07T14:00:55.387 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_background_compactions: -1 2024-04-07T14:00:55.387 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_subcompactions: 1 2024-04-07T14:00:55.387 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-07T14:00:55.387 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-07T14:00:55.387 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_total_wal_size: 0 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_open_files: -1 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bytes_per_sync: 0 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-07T14:00:55.388 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_readahead_size: 0 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_background_flushes: -1 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Compression algorithms supported: 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kZSTD supported: 0 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kXpressCompression supported: 0 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kLZ4HCCompression supported: 1 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kLZ4Compression supported: 1 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kBZip2Compression supported: 0 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kZlibCompression supported: 1 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: kSnappyCompression supported: 1 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.merge_operator: 2024-04-07T14:00:55.389 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_filter: None 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_filter_factory: None 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.sst_partitioner_factory: None 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ad977a9d00) 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks: 1 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: pin_top_level_index_and_filter: 1 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_type: 0 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_index_type: 0 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_shortening: 1 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: hash_index_allow_collision: 1 2024-04-07T14:00:55.390 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: checksum: 1 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: no_block_cache: 0 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache: 0x55ad97814170 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_name: BinnedLRUCache 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_options: 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: capacity : 536870912 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: num_shard_bits : 4 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: strict_capacity_limit : 0 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: high_pri_pool_ratio: 0.000 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_cache_compressed: (nil) 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: persistent_cache: (nil) 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size: 4096 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_size_deviation: 10 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_restart_interval: 16 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: index_block_restart_interval: 1 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: metadata_block_size: 4096 2024-04-07T14:00:55.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: partition_filters: 0 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: use_delta_encoding: 1 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: whole_key_filtering: 1 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: verify_compression: 0 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: read_amp_bytes_per_bit: 0 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: format_version: 4 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_index_compression: 1 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: block_align: 0 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.write_buffer_size: 33554432 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_write_buffer_number: 2 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression: NoCompression 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression: Disabled 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.prefix_extractor: nullptr 2024-04-07T14:00:55.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.num_levels: 7 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.level: 32767 2024-04-07T14:00:55.393 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.strategy: 0 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compression_opts.enabled: false 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.target_file_size_base: 67108864 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-07T14:00:55.394 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.arena_block_size: 4194304 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.disable_auto_compactions: 0 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-07T14:00:55.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.table_properties_collectors: 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.inplace_update_support: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.bloom_locality: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.max_successive_merges: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.paranoid_file_checks: 0 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.force_consistency_checks: 1 2024-04-07T14:00:55.396 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.report_bg_io_stats: 0 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.ttl: 2592000 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.enable_blob_files: false 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.min_blob_size: 0 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.blob_file_size: 268435456 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi111/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 5 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/version_set.cc:4083] Creating manifest 13 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498455218044, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #10 mode 2 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498455218722, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 79417, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 77681, "index_size": 224, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9884, "raw_average_key_size": 47, "raw_value_size": 71990, "raw_average_value_size": 344, "num_data_blocks": 10, "num_entries": 209, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712498455, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "9ac3442f-d41b-4589-ab3b-2b77ebd7554e", "db_session_id": "POPIE55MGO7T4SRVE0YQ"}} 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/version_set.cc:4083] Creating manifest 15 2024-04-07T14:00:55.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498455219080, "job": 1, "event": "recovery_finished"} 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi111/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x55ad977faa80 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: DB pointer 0x55ad9780a000 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** DB Stats ** 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 2/0 79.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 79.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T14:00:55.399 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 26.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 26.03 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: L0 2/0 79.35 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Sum 2/0 79.35 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-07T14:00:55.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** Compaction Stats [default] ** 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 167.2 0.00 0.00 1 0.000 0 0 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Cumulative compaction: 0.00 GB write, 25.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T14:00:55.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: starting mon.smithi111 rank 0 at public addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] at bind addrs [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi111 fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???) e1 preinit fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).mds e1 new map 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).mds e1 print_map 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: e1 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: legacy client fscid: -1 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout: No filesystems configured 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:00:55.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expand map: {default=false} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta from 'false' to 'false' 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expanded map: {default=false} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expand map: {default=info} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta from 'info' to 'info' 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expanded map: {default=info} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expand map: {default=daemon} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta from 'daemon' to 'daemon' 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expanded map: {default=daemon} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expand map: {default=debug} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta from 'debug' to 'debug' 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: expand_channel_meta expanded map: {default=debug} 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 bash[24784]: 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 systemd[1]: Started Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4. 2024-04-07T14:00:55.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mon.smithi111 is new leader, mons smithi111 in quorum (ranks 0) 2024-04-07T14:00:55.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: monmap e1: 1 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0]} 2024-04-07T14:00:55.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: fsmap 2024-04-07T14:00:55.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: osdmap e1: 0 total, 0 up, 0 in 2024-04-07T14:00:55.404 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:55 smithi111 ceph-mon[24802]: mgrmap e1: no daemons active 2024-04-07T14:00:56.389 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote config to /etc/ceph/ceph.conf 2024-04-07T14:00:56.389 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:00:56.390 INFO:teuthology.orchestra.run.smithi111.stderr:Creating mgr... 2024-04-07T14:00:56.390 INFO:teuthology.orchestra.run.smithi111.stderr:Verifying port 9283 ... 2024-04-07T14:00:56.556 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Failed to reset failed state of unit ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mgr.smithi111.rtrlso.service: Unit ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mgr.smithi111.rtrlso.service not loaded. 2024-04-07T14:00:56.563 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4.target.wants/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mgr.smithi111.rtrlso.service → /etc/systemd/system/ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@.service. 2024-04-07T14:00:56.929 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:56 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1544058534' entity='client.admin' 2024-04-07T14:00:57.227 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-04-07T14:00:57.234 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:00:57.234 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-04-07T14:00:57.234 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to enable service . firewalld.service is not available 2024-04-07T14:00:57.242 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-04-07T14:00:57.249 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:00:57.250 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-04-07T14:00:57.253 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-04-07T14:00:57.254 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr to start... 2024-04-07T14:00:57.254 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr... 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "2bfa0756-f4e7-11ee-b649-cb9ed24678a4", 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T14:00:57.789 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 2, 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T14:00:57.790 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T14:00:57.791 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": false, 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "nfs", 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:00:57.792 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-04-07T14:00:50.660599+0000", 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T14:00:57.793 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:00:58.225 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:00:57 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/939661801' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T14:00:58.289 INFO:teuthology.orchestra.run.smithi111.stderr:mgr not available, waiting (1/15)... 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "2bfa0756-f4e7-11ee-b649-cb9ed24678a4", 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T14:01:00.859 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 5, 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T14:01:00.860 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T14:01:00.861 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T14:01:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T14:01:00.864 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T14:01:00.865 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": false, 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "nfs", 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-04-07T14:00:50.660599+0000", 2024-04-07T14:01:00.866 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:01:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T14:01:00.867 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:01.401 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:01 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2726775502' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T14:01:01.428 INFO:teuthology.orchestra.run.smithi111.stderr:mgr not available, waiting (2/15)... 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: Activating manager daemon smithi111.rtrlso 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: mgrmap e2: smithi111.rtrlso(active, starting, since 0.0026941s) 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: Manager daemon smithi111.rtrlso is now available 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:01:03.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:03.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:03.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:03 smithi111 ceph-mon[24802]: from='mgr.14100 172.21.15.111:0/573093519' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:05.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:05 smithi111 ceph-mon[24802]: mgrmap e3: smithi111.rtrlso(active, since 1.00655s) 2024-04-07T14:01:07.181 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:07 smithi111 ceph-mon[24802]: mgrmap e4: smithi111.rtrlso(active, since 3s) 2024-04-07T14:01:07.774 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:01:07.776 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:07.776 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsid": "2bfa0756-f4e7-11ee-b649-cb9ed24678a4", 2024-04-07T14:01:07.776 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "health": { 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "checks": {}, 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mutes": [] 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum": [ 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 0 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_names": [ 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "smithi111" 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "quorum_age": 11, 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "monmap": { 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:07.777 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_mons": 1 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osdmap": { 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_osds": 0, 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-04-07T14:01:07.778 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgmap": { 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_pools": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_objects": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-04-07T14:01:07.782 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "bytes_total": 0 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "fsmap": { 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "by_rank": [], 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "up:standby": 0 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap": { 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modules": [ 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "iostat", 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "nfs", 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "restful" 2024-04-07T14:01:07.783 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ], 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "servicemap": { 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 1, 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "modified": "2024-04-07T14:00:50.660599+0000", 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "services": {} 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: }, 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "progress_events": {} 2024-04-07T14:01:07.784 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:08.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:08 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1356620506' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-07T14:01:10.247 INFO:teuthology.orchestra.run.smithi111.stderr:mgr is available 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [global] 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: fsid = 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-04-07T14:01:10.839 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [mgr] 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: [osd] 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-04-07T14:01:10.840 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-04-07T14:01:11.441 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:11 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2389719769' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-07T14:01:11.443 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling cephadm module... 2024-04-07T14:01:12.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:12 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1998722710' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-07T14:01:13.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:13 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1998722710' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-07T14:01:13.426 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:13 smithi111 ceph-mon[24802]: mgrmap e5: smithi111.rtrlso(active, since 9s) 2024-04-07T14:01:13.497 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:13.498 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 5, 2024-04-07T14:01:13.498 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-04-07T14:01:13.498 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "active_name": "smithi111.rtrlso", 2024-04-07T14:01:13.498 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-07T14:01:13.498 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:14.175 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for the mgr to restart... 2024-04-07T14:01:14.175 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr epoch 5... 2024-04-07T14:01:14.311 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:14 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3999688465' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-07T14:01:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:18 smithi111 ceph-mon[24802]: Active manager daemon smithi111.rtrlso restarted 2024-04-07T14:01:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:18 smithi111 ceph-mon[24802]: Activating manager daemon smithi111.rtrlso 2024-04-07T14:01:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:18 smithi111 ceph-mon[24802]: osdmap e2: 0 total, 0 up, 0 in 2024-04-07T14:01:19.170 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-04-07T14:01:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "initialized": true 2024-04-07T14:01:19.171 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: mgrmap e6: smithi111.rtrlso(active, starting, since 0.0526236s) 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: Manager daemon smithi111.rtrlso is now available 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:19.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:19.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:19.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:19.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:01:19.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:19 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:01:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:mgr epoch 5 is available 2024-04-07T14:01:19.703 INFO:teuthology.orchestra.run.smithi111.stderr:Setting orchestrator backend to cephadm... 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:19] ENGINE Bus STARTING 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: mgrmap e7: smithi111.rtrlso(active, since 1.05761s) 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: from='client.14122 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:19] ENGINE Serving on https://172.21.15.111:7150 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:19] ENGINE Bus STARTED 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:20.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:20 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:21.391 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: value unchanged 2024-04-07T14:01:21.475 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:21 smithi111 ceph-mon[24802]: mgrmap e8: smithi111.rtrlso(active, since 2s) 2024-04-07T14:01:21.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:21 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:21.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:21 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:21.940 INFO:teuthology.orchestra.run.smithi111.stderr:Generating ssh key... 2024-04-07T14:01:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:22 smithi111 ceph-mon[24802]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:23.492 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:23 smithi111 ceph-mon[24802]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:23.492 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:23 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:23.492 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:23 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:24.126 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgCHO66z05kLB2xaHf6Psos6hdsKW/qXDSUAZd4+ffEnhID2eqWQGrgoDYUq2lBAS+QfKT6GjxCwrCmf5EEZ7UMnYKqX5Aaf2xkS21DOfNuygwCmfR71tk6fYmJIQ3rQGtHQ9Sb8ds3tG1Lg87TNXwH7z3kW6PKx1QVG4HsABQ2DiJzrJayD5hoqULYCFhpRHI0EFlHjezc5xC+HJ6oeQIf6GwveW6QmgzMdBsTTeK2/MMqyodW/SMVfGIS9mI4Qd8vwh3ob8F7MOdYXNeft5IQIKLlDVcXG9ymevq/PNmA0ZrlAuC+ZPx6OZLZZvKjPEaC6qlsvrERAwW2loJ7lCNd9D+UpmVBArTelyOOwXnOuIn3GTbVB37OflwwfvhCccfmGhkZZtGSbA2idtw3cu1HZtr+4wL+1KOWPQyJUg3CxLmS3MfQL+3Cpfdi1eiB/XATL78UY4vUl05HiSX2eZnX//SeruIeX8mIWK1NJ8QIEWA+jlRCjsgI6dL38XDldU= ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:01:24.475 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:24 smithi111 ceph-mon[24802]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:24.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:24 smithi111 ceph-mon[24802]: Generating ssh key... 2024-04-07T14:01:24.669 INFO:teuthology.orchestra.run.smithi111.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-07T14:01:24.670 INFO:teuthology.orchestra.run.smithi111.stderr:Adding key to root@localhost authorized_keys... 2024-04-07T14:01:24.670 INFO:teuthology.orchestra.run.smithi111.stderr:Adding host smithi111... 2024-04-07T14:01:25.558 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:25 smithi111 ceph-mon[24802]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:26.238 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Added host 'smithi111' with addr '172.21.15.111' 2024-04-07T14:01:26.565 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:26 smithi111 ceph-mon[24802]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi111", "addr": "172.21.15.111", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:26.566 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:26 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:26.842 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying mon service with default placement... 2024-04-07T14:01:27.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:27 smithi111 ceph-mon[24802]: Deploying cephadm binary to smithi111 2024-04-07T14:01:27.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:27 smithi111 ceph-mon[24802]: Added host smithi111 2024-04-07T14:01:27.436 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:27 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:27.436 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled mon update... 2024-04-07T14:01:28.107 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying mgr service with default placement... 2024-04-07T14:01:28.518 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:28 smithi111 ceph-mon[24802]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:28.518 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:28 smithi111 ceph-mon[24802]: Saving service mon spec with placement count:5 2024-04-07T14:01:28.518 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:28 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:28.817 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-04-07T14:01:29.725 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying crash service with default placement... 2024-04-07T14:01:29.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:29 smithi111 ceph-mon[24802]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:29.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:29 smithi111 ceph-mon[24802]: Saving service mgr spec with placement count:2 2024-04-07T14:01:29.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:29 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:30.822 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled crash update... 2024-04-07T14:01:31.420 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying prometheus service with default placement... 2024-04-07T14:01:31.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:31 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:31.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:31 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:31.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:31 smithi111 ceph-mon[24802]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:31.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:31 smithi111 ceph-mon[24802]: Saving service crash spec with placement * 2024-04-07T14:01:31.656 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:31 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:32.088 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-04-07T14:01:33.209 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:33 smithi111 ceph-mon[24802]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:33.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:33 smithi111 ceph-mon[24802]: Saving service prometheus spec with placement count:1 2024-04-07T14:01:33.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:33 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:33.210 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:33 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:33.212 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying grafana service with default placement... 2024-04-07T14:01:33.863 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-04-07T14:01:34.461 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying node-exporter service with default placement... 2024-04-07T14:01:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:34 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:34 smithi111 ceph-mon[24802]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:34 smithi111 ceph-mon[24802]: Saving service grafana spec with placement count:1 2024-04-07T14:01:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:34 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:35.086 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-04-07T14:01:36.244 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:36 smithi111 ceph-mon[24802]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:36.244 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:36 smithi111 ceph-mon[24802]: Saving service node-exporter spec with placement * 2024-04-07T14:01:36.244 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:36 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:36.539 INFO:teuthology.orchestra.run.smithi111.stderr:Deploying alertmanager service with default placement... 2024-04-07T14:01:37.175 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-04-07T14:01:38.475 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:38 smithi111 ceph-mon[24802]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:38.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:38 smithi111 ceph-mon[24802]: Saving service alertmanager spec with placement count:1 2024-04-07T14:01:38.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:38 smithi111 ceph-mon[24802]: from='mgr.14118 172.21.15.111:0/3794407309' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:39.362 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:39 smithi111 ceph-mon[24802]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:01:39.362 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:39 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2351046251' entity='client.admin' 2024-04-07T14:01:40.309 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling the dashboard module... 2024-04-07T14:01:40.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:40 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/317003032' entity='client.admin' 2024-04-07T14:01:40.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:40 smithi111 ceph-mon[24802]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:01:42.079 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:41 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3516596106' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-07T14:01:42.930 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:42 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3516596106' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-07T14:01:42.930 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:42 smithi111 ceph-mon[24802]: mgrmap e9: smithi111.rtrlso(active, since 23s) 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "epoch": 9, 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "available": true, 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "active_name": "smithi111.rtrlso", 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "num_standby": 0 2024-04-07T14:01:43.021 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:43.673 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for the mgr to restart... 2024-04-07T14:01:43.673 INFO:teuthology.orchestra.run.smithi111.stderr:Waiting for mgr epoch 9... 2024-04-07T14:01:43.811 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:43 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1989799420' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-07T14:01:48.704 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: { 2024-04-07T14:01:48.704 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2024-04-07T14:01:48.704 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: "initialized": true 2024-04-07T14:01:48.704 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: } 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: Active manager daemon smithi111.rtrlso restarted 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: Activating manager daemon smithi111.rtrlso 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: osdmap e3: 0 total, 0 up, 0 in 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: mgrmap e10: smithi111.rtrlso(active, starting, since 0.0553314s) 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: Manager daemon smithi111.rtrlso is now available 2024-04-07T14:01:48.893 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:48.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:01:48.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:01:48.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:01:48.894 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:48 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:mgr epoch 9 is available 2024-04-07T14:01:49.515 INFO:teuthology.orchestra.run.smithi111.stderr:Generating a dashboard self-signed certificate... 2024-04-07T14:01:49.792 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:48] ENGINE Bus STARTING 2024-04-07T14:01:49.792 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:48] ENGINE Serving on https://172.21.15.111:7150 2024-04-07T14:01:49.792 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: [07/Apr/2024:14:01:48] ENGINE Bus STARTED 2024-04-07T14:01:49.792 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-07T14:01:49.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: mgrmap e11: smithi111.rtrlso(active, since 1.05799s) 2024-04-07T14:01:49.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-07T14:01:49.793 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:49 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:50.298 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: Self-signed certificate created 2024-04-07T14:01:50.872 INFO:teuthology.orchestra.run.smithi111.stderr:Creating initial admin user... 2024-04-07T14:01:51.281 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:51 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:51.281 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:51 smithi111 ceph-mon[24802]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:51.281 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:51 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:51.281 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:51 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:51.281 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:51 smithi111 ceph-mon[24802]: mgrmap e12: smithi111.rtrlso(active, since 2s) 2024-04-07T14:01:51.750 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$o/2V5xnUWkn3uYMlgXPOluthBdvm8hiHpCAhGfiJOBpzH26LIMhlK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1712498511, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-07T14:01:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:52 smithi111 ceph-mon[24802]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:01:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:52 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:01:53.010 INFO:teuthology.orchestra.run.smithi111.stderr:Fetching dashboard port number... 2024-04-07T14:01:53.634 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: 8443 2024-04-07T14:01:53.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:53 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/665477997' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-07T14:01:54.425 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: disabled 2024-04-07T14:01:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:systemctl: inactive 2024-04-07T14:01:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:firewalld.service is not enabled 2024-04-07T14:01:54.433 INFO:teuthology.orchestra.run.smithi111.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr:Ceph Dashboard is now available at: 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr: URL: https://smithi111.front.sepia.ceph.com:8443/ 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr: User: admin 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr: Password: ilhgtht7zi 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:54.436 INFO:teuthology.orchestra.run.smithi111.stderr:Enabling autotune for osd_memory_target 2024-04-07T14:01:54.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:54 smithi111 ceph-mon[24802]: mgrmap e13: smithi111.rtrlso(active, since 6s) 2024-04-07T14:01:56.374 INFO:teuthology.orchestra.run.smithi111.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr:You can access the Ceph CLI with: 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr:Please consider enabling telemetry to help improve Ceph: 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.980 INFO:teuthology.orchestra.run.smithi111.stderr: ceph telemetry on 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr:For more information see: 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:01:56.981 INFO:teuthology.orchestra.run.smithi111.stderr:Bootstrap complete. 2024-04-07T14:01:57.017 INFO:tasks.cephadm:Fetching config... 2024-04-07T14:01:57.017 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:01:57.017 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-07T14:01:57.033 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-07T14:01:57.033 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:01:57.033 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-07T14:01:57.088 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-07T14:01:57.088 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:01:57.088 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/keyring of=/dev/stdout 2024-04-07T14:01:57.154 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-07T14:01:57.154 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:01:57.154 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-07T14:01:57.209 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-07T14:01:57.209 DEBUG:teuthology.orchestra.run.smithi111:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgCHO66z05kLB2xaHf6Psos6hdsKW/qXDSUAZd4+ffEnhID2eqWQGrgoDYUq2lBAS+QfKT6GjxCwrCmf5EEZ7UMnYKqX5Aaf2xkS21DOfNuygwCmfR71tk6fYmJIQ3rQGtHQ9Sb8ds3tG1Lg87TNXwH7z3kW6PKx1QVG4HsABQ2DiJzrJayD5hoqULYCFhpRHI0EFlHjezc5xC+HJ6oeQIf6GwveW6QmgzMdBsTTeK2/MMqyodW/SMVfGIS9mI4Qd8vwh3ob8F7MOdYXNeft5IQIKLlDVcXG9ymevq/PNmA0ZrlAuC+ZPx6OZLZZvKjPEaC6qlsvrERAwW2loJ7lCNd9D+UpmVBArTelyOOwXnOuIn3GTbVB37OflwwfvhCccfmGhkZZtGSbA2idtw3cu1HZtr+4wL+1KOWPQyJUg3CxLmS3MfQL+3Cpfdi1eiB/XATL78UY4vUl05HiSX2eZnX//SeruIeX8mIWK1NJ8QIEWA+jlRCjsgI6dL38XDldU= ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-07T14:01:57.287 INFO:teuthology.orchestra.run.smithi111.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgCHO66z05kLB2xaHf6Psos6hdsKW/qXDSUAZd4+ffEnhID2eqWQGrgoDYUq2lBAS+QfKT6GjxCwrCmf5EEZ7UMnYKqX5Aaf2xkS21DOfNuygwCmfR71tk6fYmJIQ3rQGtHQ9Sb8ds3tG1Lg87TNXwH7z3kW6PKx1QVG4HsABQ2DiJzrJayD5hoqULYCFhpRHI0EFlHjezc5xC+HJ6oeQIf6GwveW6QmgzMdBsTTeK2/MMqyodW/SMVfGIS9mI4Qd8vwh3ob8F7MOdYXNeft5IQIKLlDVcXG9ymevq/PNmA0ZrlAuC+ZPx6OZLZZvKjPEaC6qlsvrERAwW2loJ7lCNd9D+UpmVBArTelyOOwXnOuIn3GTbVB37OflwwfvhCccfmGhkZZtGSbA2idtw3cu1HZtr+4wL+1KOWPQyJUg3CxLmS3MfQL+3Cpfdi1eiB/XATL78UY4vUl05HiSX2eZnX//SeruIeX8mIWK1NJ8QIEWA+jlRCjsgI6dL38XDldU= ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:01:57.302 DEBUG:teuthology.orchestra.run.smithi145:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgCHO66z05kLB2xaHf6Psos6hdsKW/qXDSUAZd4+ffEnhID2eqWQGrgoDYUq2lBAS+QfKT6GjxCwrCmf5EEZ7UMnYKqX5Aaf2xkS21DOfNuygwCmfR71tk6fYmJIQ3rQGtHQ9Sb8ds3tG1Lg87TNXwH7z3kW6PKx1QVG4HsABQ2DiJzrJayD5hoqULYCFhpRHI0EFlHjezc5xC+HJ6oeQIf6GwveW6QmgzMdBsTTeK2/MMqyodW/SMVfGIS9mI4Qd8vwh3ob8F7MOdYXNeft5IQIKLlDVcXG9ymevq/PNmA0ZrlAuC+ZPx6OZLZZvKjPEaC6qlsvrERAwW2loJ7lCNd9D+UpmVBArTelyOOwXnOuIn3GTbVB37OflwwfvhCccfmGhkZZtGSbA2idtw3cu1HZtr+4wL+1KOWPQyJUg3CxLmS3MfQL+3Cpfdi1eiB/XATL78UY4vUl05HiSX2eZnX//SeruIeX8mIWK1NJ8QIEWA+jlRCjsgI6dL38XDldU= ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-07T14:01:57.350 INFO:teuthology.orchestra.run.smithi145.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCgCHO66z05kLB2xaHf6Psos6hdsKW/qXDSUAZd4+ffEnhID2eqWQGrgoDYUq2lBAS+QfKT6GjxCwrCmf5EEZ7UMnYKqX5Aaf2xkS21DOfNuygwCmfR71tk6fYmJIQ3rQGtHQ9Sb8ds3tG1Lg87TNXwH7z3kW6PKx1QVG4HsABQ2DiJzrJayD5hoqULYCFhpRHI0EFlHjezc5xC+HJ6oeQIf6GwveW6QmgzMdBsTTeK2/MMqyodW/SMVfGIS9mI4Qd8vwh3ob8F7MOdYXNeft5IQIKLlDVcXG9ymevq/PNmA0ZrlAuC+ZPx6OZLZZvKjPEaC6qlsvrERAwW2loJ7lCNd9D+UpmVBArTelyOOwXnOuIn3GTbVB37OflwwfvhCccfmGhkZZtGSbA2idtw3cu1HZtr+4wL+1KOWPQyJUg3CxLmS3MfQL+3Cpfdi1eiB/XATL78UY4vUl05HiSX2eZnX//SeruIeX8mIWK1NJ8QIEWA+jlRCjsgI6dL38XDldU= ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:01:57.366 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-07T14:01:57.676 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:01:57 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3174726156' entity='client.admin' 2024-04-07T14:02:00.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:00 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:00.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:00 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:02:00.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:00 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:00.221 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:00 smithi111 ceph-mon[24802]: Deploying daemon alertmanager.smithi111 on smithi111 2024-04-07T14:02:01.545 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-07T14:02:01.545 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-07T14:02:02.091 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:01 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/485834062' entity='client.admin' 2024-04-07T14:02:04.991 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi145 2024-04-07T14:02:04.992 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T14:02:04.992 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.conf 2024-04-07T14:02:05.011 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T14:02:05.011 DEBUG:teuthology.orchestra.run.smithi145:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:02:05.069 INFO:tasks.cephadm:Adding host smithi145 to orchestrator... 2024-04-07T14:02:05.070 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch host add smithi145 2024-04-07T14:02:05.097 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:04 smithi111 ceph-mon[24802]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:02:05.097 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:04 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:08.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:08 smithi111 ceph-mon[24802]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:11.084 INFO:teuthology.orchestra.run.smithi111.stdout:Added host 'smithi145' with addr '172.21.15.145' 2024-04-07T14:02:11.084 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:10 smithi111 ceph-mon[24802]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:11.084 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:10 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:11.084 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:10 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:02:11.084 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:10 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-07T14:02:11.084 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:10 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:12.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:11 smithi111 ceph-mon[24802]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:02:12.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:11 smithi111 ceph-mon[24802]: Deploying cephadm binary to smithi145 2024-04-07T14:02:12.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:11 smithi111 ceph-mon[24802]: Deploying daemon crash.smithi111 on smithi111 2024-04-07T14:02:12.007 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:11 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:12.549 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch host ls --format=json 2024-04-07T14:02:12.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:12 smithi111 ceph-mon[24802]: Added host smithi145 2024-04-07T14:02:12.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:12 smithi111 ceph-mon[24802]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:12.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:12 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: Deploying daemon grafana.smithi111 on smithi111 2024-04-07T14:02:14.452 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:14 smithi111 ceph-mon[24802]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:14.962 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:02:14.963 INFO:teuthology.orchestra.run.smithi111.stdout:[{"addr": "172.21.15.111", "hostname": "smithi111", "labels": [], "status": ""}, {"addr": "172.21.15.145", "hostname": "smithi145", "labels": [], "status": ""}] 2024-04-07T14:02:15.501 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-07T14:02:15.501 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd crush tunables default 2024-04-07T14:02:16.475 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:16 smithi111 ceph-mon[24802]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:02:16.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:16 smithi111 ceph-mon[24802]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:17.660 INFO:teuthology.orchestra.run.smithi111.stderr:adjusted tunables profile to default 2024-04-07T14:02:17.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:17 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3807030048' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-07T14:02:18.349 INFO:tasks.cephadm:Adding mon.smithi111 on smithi111 2024-04-07T14:02:18.349 INFO:tasks.cephadm:Adding mon.smithi145 on smithi145 2024-04-07T14:02:18.349 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch apply mon '2;smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145' 2024-04-07T14:02:18.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:18 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3807030048' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-07T14:02:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:18 smithi111 ceph-mon[24802]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T14:02:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:18 smithi111 ceph-mon[24802]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:18 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:20.055 INFO:teuthology.orchestra.run.smithi145.stdout:Scheduled mon update... 2024-04-07T14:02:20.605 DEBUG:teuthology.orchestra.run.smithi145:mon.smithi145> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi145.service 2024-04-07T14:02:20.608 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:20.608 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:21.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:20 smithi111 ceph-mon[24802]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:21.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:20 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:22.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:21 smithi111 ceph-mon[24802]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:02:22.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:21 smithi111 ceph-mon[24802]: Saving service mon spec with placement smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145;count:2 2024-04-07T14:02:22.282 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:22.283 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:22.284 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:23.225 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:22 smithi111 ceph-mon[24802]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:23.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:22 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/1078210066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:23.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:23.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:25.022 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:24 smithi111 ceph-mon[24802]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:25.742 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:25.742 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:25.744 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:26 smithi111 ceph-mon[24802]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:26 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/3720393394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:27.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:27.227 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:28 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:28 smithi111 ceph-mon[24802]: Deploying daemon node-exporter.smithi111 on smithi111 2024-04-07T14:02:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:28 smithi111 ceph-mon[24802]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:28 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:29.023 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:29.023 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:29.024 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:29.788 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:29 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/3862953947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:30.538 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:30.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:30.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:30 smithi111 ceph-mon[24802]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:31.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:31 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:31.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:31 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T14:02:32.343 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:32.343 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:32.345 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:32 smithi111 ceph-mon[24802]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T14:02:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:32 smithi111 ceph-mon[24802]: mgrmap e14: smithi111.rtrlso(active, since 44s) 2024-04-07T14:02:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:32 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/3461196789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:33.823 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:33.824 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:35.700 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:35.701 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:35.702 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:36.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:36 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/3131885157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:37.236 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:37.236 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:37.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:37 smithi111 ceph-mon[24802]: Active manager daemon smithi111.rtrlso restarted 2024-04-07T14:02:37.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:37 smithi111 ceph-mon[24802]: Activating manager daemon smithi111.rtrlso 2024-04-07T14:02:37.852 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:37 smithi111 ceph-mon[24802]: osdmap e5: 0 total, 0 up, 0 in 2024-04-07T14:02:38.704 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: mgrmap e15: smithi111.rtrlso(active, starting, since 0.0525337s) 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: Manager daemon smithi111.rtrlso is now available 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:38.705 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:39.122 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:39.122 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:39.124 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:39.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: [07/Apr/2024:14:02:38] ENGINE Bus STARTING 2024-04-07T14:02:39.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: [07/Apr/2024:14:02:38] ENGINE Serving on https://172.21.15.111:7150 2024-04-07T14:02:39.605 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: [07/Apr/2024:14:02:38] ENGINE Bus STARTED 2024-04-07T14:02:39.606 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: mgrmap e16: smithi111.rtrlso(active, since 1.05768s) 2024-04-07T14:02:39.606 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:39.606 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:39 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/1101331884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:40.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:40.668 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:40.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:40.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:40 smithi111 ceph-mon[24802]: mgrmap e17: smithi111.rtrlso(active, since 2s) 2024-04-07T14:02:42.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:42.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:42.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:02:43.584 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:43.584 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:43.586 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:02:43.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-07T14:02:43.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:44.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:44 smithi111 ceph-mon[24802]: Deploying daemon crash.smithi145 on smithi145 2024-04-07T14:02:44.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:44 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/1434864205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:45.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:45.947 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:47.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:46 smithi111 ceph-mon[24802]: Deploying daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:02:48.938 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:48.938 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:48.940 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:48.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:49.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:49 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/4048410098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:49.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:49 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:49.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:49 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:02:49.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:49 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:50.831 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:50.831 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:50 smithi111 ceph-mon[24802]: Deploying daemon mon.smithi145 on smithi145 2024-04-07T14:02:53.538 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:53 smithi145 podman[26737]: 2024-04-07 14:02:53.419389348 +0000 UTC m=+0.020065408 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:02:53.888 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:02:53.889 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":1,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:00:47.823097Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-07T14:02:53.890 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 1 2024-04-07T14:02:53.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:53 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.166 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:53 smithi145 podman[26737]: 2024-04-07 14:02:53.797248445 +0000 UTC m=+0.397924531 container create c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145, architecture=x86_64, distribution-scope=public, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vendor=Red Hat, Inc., io.openshift.expose-services=, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, version=8, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, release=754, ceph=True, vcs-type=git, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T14:02:54.167 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 podman[26737]: 2024-04-07 14:02:54.167247244 +0000 UTC m=+0.767923286 container init c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, io.openshift.expose-services=, distribution-scope=public, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=-17.2.0, GIT_CLEAN=True, architecture=x86_64, io.buildah.version=1.19.8, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, release=754, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T14:02:54.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 podman[26737]: 2024-04-07 14:02:54.171290552 +0000 UTC m=+0.771966593 container start c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, version=8, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, release=754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, name=centos-stream, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , ceph=True, vendor=Red Hat, Inc., RELEASE=HEAD, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870) 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: set uid:gid to 167:167 (ceph:ceph) 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable), process ceph-mon, pid 2 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pidfile_write: ignore empty --pid-file 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: load: jerasure load: lrc 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: RocksDB version: 6.15.5 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Compile date Apr 18 2022 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: DB SUMMARY 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: DB Session ID: VBGTUFLN7B534R0XQEP9 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: CURRENT file: CURRENT 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: IDENTITY file: IDENTITY 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi145/store.db dir, Total Num: 0, files: 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi145/store.db: 000004.log size: 511 ; 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.error_if_exists: 0 2024-04-07T14:02:54.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.create_if_missing: 0 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.paranoid_checks: 1 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.env: 0x5564ea8c6860 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.fs: Posix File System 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.info_log: 0x5564eb713fa0 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_file_opening_threads: 16 2024-04-07T14:02:54.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.statistics: (nil) 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.use_fsync: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_log_file_size: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.keep_log_file_num: 1000 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.recycle_log_file_num: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_fallocate: 1 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_mmap_reads: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_mmap_writes: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.use_direct_reads: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.create_missing_column_families: 0 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.db_log_dir: 2024-04-07T14:02:54.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi145/store.db 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.advise_random_on_open: 1 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.db_write_buffer_size: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.write_buffer_manager: 0x5564eb8042d0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.rate_limiter: (nil) 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-07T14:02:54.527 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.wal_recovery_mode: 2 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.enable_thread_tracking: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.enable_pipelined_write: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.unordered_write: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.row_cache: None 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.wal_filter: None 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_ingest_behind: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.preserve_deletes: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.two_write_queues: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.manual_wal_flush: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.atomic_flush: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-07T14:02:54.528 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.log_readahead_size: 0 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.best_efforts_recovery: 0 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.allow_data_in_errors: 0 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.db_host_id: __hostname__ 2024-04-07T14:02:54.529 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_background_jobs: 2 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_background_compactions: -1 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_subcompactions: 1 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_total_wal_size: 0 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-07T14:02:54.530 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_open_files: -1 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bytes_per_sync: 0 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_readahead_size: 0 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_background_flushes: -1 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Compression algorithms supported: 2024-04-07T14:02:54.531 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kZSTD supported: 0 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kXpressCompression supported: 0 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kLZ4HCCompression supported: 1 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kLZ4Compression supported: 1 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kBZip2Compression supported: 0 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kZlibCompression supported: 1 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: kSnappyCompression supported: 1 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/version_set.cc:4725] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi145/store.db/MANIFEST-000003 2024-04-07T14:02:54.532 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/column_family.cc:597] --------------- Options for column family [default]: 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.merge_operator: 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_filter: None 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_filter_factory: None 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.sst_partitioner_factory: None 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5564eb6dfd00) 2024-04-07T14:02:54.533 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: cache_index_and_filter_blocks: 1 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: pin_top_level_index_and_filter: 1 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: index_type: 0 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: data_block_index_type: 0 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: index_shortening: 1 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: hash_index_allow_collision: 1 2024-04-07T14:02:54.534 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: checksum: 1 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: no_block_cache: 0 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_cache: 0x5564eb74a170 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_cache_name: BinnedLRUCache 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_cache_options: 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: capacity : 536870912 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: num_shard_bits : 4 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: strict_capacity_limit : 0 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: high_pri_pool_ratio: 0.000 2024-04-07T14:02:54.535 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_cache_compressed: (nil) 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: persistent_cache: (nil) 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_size: 4096 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_size_deviation: 10 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_restart_interval: 16 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: index_block_restart_interval: 1 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: metadata_block_size: 4096 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: partition_filters: 0 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: use_delta_encoding: 1 2024-04-07T14:02:54.536 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: whole_key_filtering: 1 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: verify_compression: 0 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: read_amp_bytes_per_bit: 0 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: format_version: 4 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: enable_index_compression: 1 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: block_align: 0 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.write_buffer_size: 33554432 2024-04-07T14:02:54.537 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_write_buffer_number: 2 2024-04-07T14:02:54.539 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression: NoCompression 2024-04-07T14:02:54.539 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression: Disabled 2024-04-07T14:02:54.539 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.prefix_extractor: nullptr 2024-04-07T14:02:54.539 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.num_levels: 7 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-07T14:02:54.540 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.level: 32767 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.strategy: 0 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-07T14:02:54.541 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compression_opts.enabled: false 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.target_file_size_base: 67108864 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-07T14:02:54.542 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-07T14:02:54.543 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.arena_block_size: 4194304 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.disable_auto_compactions: 0 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.table_properties_collectors: 2024-04-07T14:02:54.544 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.inplace_update_support: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.bloom_locality: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.max_successive_merges: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.paranoid_file_checks: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.force_consistency_checks: 1 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.report_bg_io_stats: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.ttl: 2592000 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.enable_blob_files: false 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.min_blob_size: 0 2024-04-07T14:02:54.545 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.blob_file_size: 268435456 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/version_set.cc:4773] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi145/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/version_set.cc:4782] Column family [default] (ID 0), log number is 0 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/version_set.cc:4083] Creating manifest 7 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498574197326, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/db_impl/db_impl_open.cc:847] Recovering log #4 mode 2 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [table/block_based/filter_policy.cc:996] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498574197630, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1712498574, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "85351c7d-3e0e-4f4d-bf3c-d0dfa06bebbe", "db_session_id": "VBGTUFLN7B534R0XQEP9"}} 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/version_set.cc:4083] Creating manifest 9 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: EVENT_LOG_v1 {"time_micros": 1712498574197940, "job": 1, "event": "recovery_finished"} 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [file/delete_scheduler.cc:73] Deleted file /var/lib/ceph/mon/ceph-smithi145/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/db_impl/db_impl_open.cc:1701] SstFileManager instance 0x5564eb730a80 2024-04-07T14:02:54.546 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: DB pointer 0x5564eb740000 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/db_impl/db_impl.cc:902] ------- DUMPING STATS ------- 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** DB Stats ** 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** Compaction Stats [default] ** 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.547 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** Compaction Stats [default] ** 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Cumulative compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Interval compaction: 0.00 GB write, 0.59 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.548 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** Compaction Stats [default] ** 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** Compaction Stats [default] ** 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.0 0.00 0.00 1 0.000 0 0 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-07T14:02:54.549 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Cumulative compaction: 0.00 GB write, 0.58 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145 does not exist in monmap, will attempt to join an existing cluster 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: using public_addr v2:172.21.15.145:0/0 -> [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: starting mon.smithi145 rank -1 at public addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] at bind addrs [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi145 fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(???) e0 preinit fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).mds e1 new map 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).mds e1 print_map 2024-04-07T14:02:54.550 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: e1 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: legacy client fscid: -1 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout: No filesystems configured 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:02:54.551 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi145", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying cephadm binary to smithi145 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon crash.smithi111 on smithi111 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Added host smithi145 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon grafana.smithi111 on smithi111 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:02:54.552 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3807030048' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3807030048' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: osdmap e4: 0 total, 0 up, 0 in 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Saving service mon spec with placement smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145;count:2 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/1078210066' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/3720393394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.553 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon node-exporter.smithi111 on smithi111 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/3862953947' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14160 172.21.15.111:0/2255581154' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mgrmap e14: smithi111.rtrlso(active, since 44s) 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/3461196789' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/3131885157' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Active manager daemon smithi111.rtrlso restarted 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Activating manager daemon smithi111.rtrlso 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: osdmap e5: 0 total, 0 up, 0 in 2024-04-07T14:02:54.554 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mgrmap e15: smithi111.rtrlso(active, starting, since 0.0525337s) 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Manager daemon smithi111.rtrlso is now available 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.555 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: [07/Apr/2024:14:02:38] ENGINE Bus STARTING 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: [07/Apr/2024:14:02:38] ENGINE Serving on https://172.21.15.111:7150 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: [07/Apr/2024:14:02:38] ENGINE Bus STARTED 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mgrmap e16: smithi111.rtrlso(active, since 1.05768s) 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/1101331884' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mgrmap e17: smithi111.rtrlso(active, since 2s) 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd/host:smithi111", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Updating smithi111:/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd/host:smithi145", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:02:54.556 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon crash.smithi145 on smithi145 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/1434864205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/4048410098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:02:54.557 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: Deploying daemon mon.smithi145 on smithi145 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 ceph-mon[26800]: mon.smithi145@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete,mon_cluster_log_to_file 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 bash[26737]: c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 2024-04-07T14:02:54.558 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:54 smithi145 systemd[1]: Started Ceph mon.smithi145 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4. 2024-04-07T14:02:55.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-04-07T14:02:55.776 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mon dump -f json 2024-04-07T14:02:59.576 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:02:59.576 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: mon.smithi111 calling monitor election 2024-04-07T14:02:59.576 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.576 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi145.qiunpb/crt"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: mon.smithi145 calling monitor election 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: mon.smithi111 is new leader, mons smithi111,smithi145 in quorum (ranks 0,1) 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi145=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: fsmap 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: osdmap e5: 0 total, 0 up, 0 in 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: mgrmap e17: smithi111.rtrlso(active, since 21s) 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: Standby manager daemon smithi145.qiunpb started 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: overall HEALTH_OK 2024-04-07T14:02:59.577 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi145.qiunpb/key"}]: dispatch 2024-04-07T14:02:59.578 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:59.578 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:02:59 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: mon.smithi111 calling monitor election 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi145.qiunpb/crt"}]: dispatch 2024-04-07T14:02:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: mon.smithi145 calling monitor election 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: mon.smithi111 is new leader, mons smithi111,smithi145 in quorum (ranks 0,1) 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: monmap e2: 2 mons at {smithi111=[v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0],smithi145=[v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0]} 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: fsmap 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: osdmap e5: 0 total, 0 up, 0 in 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: mgrmap e17: smithi111.rtrlso(active, since 21s) 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: Standby manager daemon smithi145.qiunpb started 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: overall HEALTH_OK 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi145.qiunpb/key"}]: dispatch 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:02:59.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:02:59 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.145:0/2837000437' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T14:03:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:00 smithi111 ceph-mon[24802]: Deploying daemon node-exporter.smithi145 on smithi145 2024-04-07T14:03:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:00 smithi111 ceph-mon[24802]: mgrmap e18: smithi111.rtrlso(active, since 21s), standbys: smithi145.qiunpb 2024-04-07T14:03:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi145.qiunpb", "id": "smithi145.qiunpb"}]: dispatch 2024-04-07T14:03:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:00 smithi111 ceph-mon[24802]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:03:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:00 smithi145 ceph-mon[26800]: Deploying daemon node-exporter.smithi145 on smithi145 2024-04-07T14:03:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:00 smithi145 ceph-mon[26800]: mgrmap e18: smithi111.rtrlso(active, since 21s), standbys: smithi145.qiunpb 2024-04-07T14:03:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr metadata", "who": "smithi145.qiunpb", "id": "smithi145.qiunpb"}]: dispatch 2024-04-07T14:03:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:00 smithi145 ceph-mon[26800]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:03:00.777 INFO:teuthology.orchestra.run.smithi145.stdout: 2024-04-07T14:03:00.777 INFO:teuthology.orchestra.run.smithi145.stdout:{"epoch":2,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","modified":"2024-04-07T14:02:54.225167Z","created":"2024-04-07T14:00:47.823097Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi111","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:3300","nonce":0},{"type":"v1","addr":"172.21.15.111:6789","nonce":0}]},"addr":"172.21.15.111:6789/0","public_addr":"172.21.15.111:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi145","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:3300","nonce":0},{"type":"v1","addr":"172.21.15.145:6789","nonce":0}]},"addr":"172.21.15.145:6789/0","public_addr":"172.21.15.145:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-07T14:03:00.778 INFO:teuthology.orchestra.run.smithi145.stderr:dumped monmap epoch 2 2024-04-07T14:03:01.690 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:01 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/4034763864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:03:01.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:01 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/4034763864' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-07T14:03:02.186 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-07T14:03:02.187 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph config generate-minimal-conf 2024-04-07T14:03:02.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:02 smithi111 ceph-mon[24802]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:02.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:02 smithi145 ceph-mon[26800]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:03.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:03 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:03.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:03 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:03 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:03 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:04.116 INFO:teuthology.orchestra.run.smithi111.stdout:# minimal ceph.conf for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:03:04.116 INFO:teuthology.orchestra.run.smithi111.stdout:[global] 2024-04-07T14:03:04.116 INFO:teuthology.orchestra.run.smithi111.stdout: fsid = 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 2024-04-07T14:03:04.116 INFO:teuthology.orchestra.run.smithi111.stdout: mon_host = [v2:172.21.15.111:3300/0,v1:172.21.15.111:6789/0] [v2:172.21.15.145:3300/0,v1:172.21.15.145:6789/0] 2024-04-07T14:03:04.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:04 smithi111 ceph-mon[24802]: Deploying daemon prometheus.smithi111 on smithi111 2024-04-07T14:03:04.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:04 smithi111 ceph-mon[24802]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:04.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:04 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1897980442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:05.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:04 smithi145 ceph-mon[26800]: Deploying daemon prometheus.smithi111 on smithi111 2024-04-07T14:03:05.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:04 smithi145 ceph-mon[26800]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:05.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:04 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1897980442' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:05.086 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-07T14:03:05.087 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:03:05.087 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.conf 2024-04-07T14:03:05.117 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:03:05.117 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:03:05.187 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T14:03:05.187 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.conf 2024-04-07T14:03:05.222 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T14:03:05.223 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:03:05.294 INFO:tasks.cephadm:Deploying OSDs... 2024-04-07T14:03:05.294 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:03:05.294 DEBUG:teuthology.orchestra.run.smithi111:> dd if=/scratch_devs of=/dev/stdout 2024-04-07T14:03:05.312 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-07T14:03:05.312 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_1 2024-04-07T14:03:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-07T14:03:05.371 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:05.371 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-07T14:03:05.371 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:05.371 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:05.372 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-07 14:02:28.191875495 +0000 2024-04-07T14:03:05.372 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-07 13:59:55.138146637 +0000 2024-04-07T14:03:05.372 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-07 13:59:55.138146637 +0000 2024-04-07T14:03:05.372 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-07 13:59:55.138146637 +0000 2024-04-07T14:03:05.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-07T14:03:05.439 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-07T14:03:05.439 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-07T14:03:05.439 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000255636 s, 2.0 MB/s 2024-04-07T14:03:05.440 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-07T14:03:05.497 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_2 2024-04-07T14:03:05.551 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-07 14:02:28.192875478 +0000 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-07 13:59:55.476140682 +0000 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-07 13:59:55.476140682 +0000 2024-04-07T14:03:05.552 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-07 13:59:55.476140682 +0000 2024-04-07T14:03:05.552 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-07T14:03:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-07T14:03:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-07T14:03:05.617 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000239417 s, 2.1 MB/s 2024-04-07T14:03:05.618 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-07T14:03:05.675 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_3 2024-04-07T14:03:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-07T14:03:05.729 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:05.729 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-07 14:02:28.192875478 +0000 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-07 13:59:55.812134762 +0000 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-07 13:59:55.812134762 +0000 2024-04-07T14:03:05.730 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-07 13:59:55.812134762 +0000 2024-04-07T14:03:05.730 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-07T14:03:05.795 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-07T14:03:05.795 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-07T14:03:05.795 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000175791 s, 2.9 MB/s 2024-04-07T14:03:05.797 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-07T14:03:05.852 DEBUG:teuthology.orchestra.run.smithi111:> stat /dev/vg_nvme/lv_4 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:Access: 2024-04-07 14:02:28.192875478 +0000 2024-04-07T14:03:05.907 INFO:teuthology.orchestra.run.smithi111.stdout:Modify: 2024-04-07 13:59:56.140128984 +0000 2024-04-07T14:03:05.908 INFO:teuthology.orchestra.run.smithi111.stdout:Change: 2024-04-07 13:59:56.140128984 +0000 2024-04-07T14:03:05.908 INFO:teuthology.orchestra.run.smithi111.stdout: Birth: 2024-04-07 13:59:56.140128984 +0000 2024-04-07T14:03:05.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-07T14:03:05.972 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records in 2024-04-07T14:03:05.972 INFO:teuthology.orchestra.run.smithi111.stderr:1+0 records out 2024-04-07T14:03:05.972 INFO:teuthology.orchestra.run.smithi111.stderr:512 bytes copied, 0.000180605 s, 2.8 MB/s 2024-04-07T14:03:05.973 DEBUG:teuthology.orchestra.run.smithi111:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-07T14:03:06.028 DEBUG:teuthology.orchestra.run.smithi145:> set -ex 2024-04-07T14:03:06.028 DEBUG:teuthology.orchestra.run.smithi145:> dd if=/scratch_devs of=/dev/stdout 2024-04-07T14:03:06.047 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-07T14:03:06.047 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_1 2024-04-07T14:03:06.104 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 668 Links: 1 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-07 14:02:59.863921339 +0000 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-07 13:59:55.077236332 +0000 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-07 13:59:55.077236332 +0000 2024-04-07T14:03:06.105 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-04-07 13:59:55.077236332 +0000 2024-04-07T14:03:06.105 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-07T14:03:06.173 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-07T14:03:06.173 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-07T14:03:06.174 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000236093 s, 2.2 MB/s 2024-04-07T14:03:06.175 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-07T14:03:06.233 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_2 2024-04-07T14:03:06.290 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-07 14:02:59.863921339 +0000 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-07 13:59:55.422228835 +0000 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-07 13:59:55.422228835 +0000 2024-04-07T14:03:06.291 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-04-07 13:59:55.422228835 +0000 2024-04-07T14:03:06.291 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-07T14:03:06.359 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-07T14:03:06.359 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-07T14:03:06.359 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000242766 s, 2.1 MB/s 2024-04-07T14:03:06.360 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-07T14:03:06.416 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_3 2024-04-07T14:03:06.471 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-07 14:02:59.863921339 +0000 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-07 13:59:55.755221599 +0000 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-07 13:59:55.755221599 +0000 2024-04-07T14:03:06.472 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-04-07 13:59:55.755221599 +0000 2024-04-07T14:03:06.472 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-07T14:03:06.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:06 smithi145 ceph-mon[26800]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:06.538 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-07T14:03:06.538 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-07T14:03:06.538 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000173214 s, 3.0 MB/s 2024-04-07T14:03:06.539 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-07T14:03:06.595 DEBUG:teuthology.orchestra.run.smithi145:> stat /dev/vg_nvme/lv_4 2024-04-07T14:03:06.650 INFO:teuthology.orchestra.run.smithi145.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-07T14:03:06.650 INFO:teuthology.orchestra.run.smithi145.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-07T14:03:06.650 INFO:teuthology.orchestra.run.smithi145.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-04-07T14:03:06.650 INFO:teuthology.orchestra.run.smithi145.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-07T14:03:06.651 INFO:teuthology.orchestra.run.smithi145.stdout:Context: system_u:object_r:device_t:s0 2024-04-07T14:03:06.651 INFO:teuthology.orchestra.run.smithi145.stdout:Access: 2024-04-07 14:02:59.864921318 +0000 2024-04-07T14:03:06.651 INFO:teuthology.orchestra.run.smithi145.stdout:Modify: 2024-04-07 13:59:56.118213711 +0000 2024-04-07T14:03:06.651 INFO:teuthology.orchestra.run.smithi145.stdout:Change: 2024-04-07 13:59:56.118213711 +0000 2024-04-07T14:03:06.651 INFO:teuthology.orchestra.run.smithi145.stdout: Birth: 2024-04-07 13:59:56.118213711 +0000 2024-04-07T14:03:06.651 DEBUG:teuthology.orchestra.run.smithi145:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-07T14:03:06.716 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records in 2024-04-07T14:03:06.717 INFO:teuthology.orchestra.run.smithi145.stderr:1+0 records out 2024-04-07T14:03:06.717 INFO:teuthology.orchestra.run.smithi145.stderr:512 bytes copied, 0.000320101 s, 1.6 MB/s 2024-04-07T14:03:06.718 DEBUG:teuthology.orchestra.run.smithi145:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-07T14:03:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:06 smithi111 ceph-mon[24802]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:06.774 INFO:tasks.cephadm:Deploying osd.0 on smithi111 with /dev/vg_nvme/lv_4... 2024-04-07T14:03:06.774 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-07T14:03:08.723 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:08 smithi145 ceph-mon[26800]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:08.723 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:08.834 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:08 smithi111 ceph-mon[24802]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:08.834 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:10.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:10 smithi111 ceph-mon[24802]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112312 s, 934 MB/s 2024-04-07T14:03:10.990 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-07T14:03:11.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:10 smithi145 ceph-mon[26800]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:12.207 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi111:vg_nvme/lv_4 2024-04-07T14:03:12.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:12 smithi111 ceph-mon[24802]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:13.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:12 smithi145 ceph-mon[26800]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:14 smithi111 ceph-mon[24802]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:15.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:14 smithi145 ceph-mon[26800]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:16.400 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:16 smithi111 ceph-mon[24802]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:16.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:16 smithi145 ceph-mon[26800]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:17.351 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:03:17.351 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:03:17.351 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:17.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:03:17.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:03:17.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:03:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:03:18.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:18.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:03:18.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:18.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:18.617 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='client.14232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:03:18.617 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:18.617 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:03:18.618 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:18.618 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:03:18.618 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:18.618 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:20.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:20.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: Updating smithi145:/etc/ceph/ceph.conf 2024-04-07T14:03:20.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:03:20.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1185178253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4937d42-740b-4c86-8497-4f8d55b1c30b"}]: dispatch 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1185178253' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4937d42-740b-4c86-8497-4f8d55b1c30b"}]': finished 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: osdmap e6: 1 total, 0 up, 1 in 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:20.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:20 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2473078837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: Updating smithi145:/etc/ceph/ceph.conf 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: Updating smithi145:/etc/ceph/ceph.client.admin.keyring 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1185178253' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4937d42-740b-4c86-8497-4f8d55b1c30b"}]: dispatch 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1185178253' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4937d42-740b-4c86-8497-4f8d55b1c30b"}]': finished 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: osdmap e6: 1 total, 0 up, 1 in 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:20.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:20 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2473078837' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:03:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:22.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:22.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:03:22.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T14:03:22.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:23.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T14:03:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:23.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:23 smithi111 ceph-mon[24802]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-07T14:03:23.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:23 smithi111 ceph-mon[24802]: Reconfiguring mon.smithi111 (unknown last config time)... 2024-04-07T14:03:23.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:23 smithi111 ceph-mon[24802]: Reconfiguring daemon mon.smithi111 on smithi111 2024-04-07T14:03:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:23 smithi145 ceph-mon[26800]: Updating smithi111:/etc/ceph/ceph.conf 2024-04-07T14:03:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:23 smithi145 ceph-mon[26800]: Reconfiguring mon.smithi111 (unknown last config time)... 2024-04-07T14:03:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:23 smithi145 ceph-mon[26800]: Reconfiguring daemon mon.smithi111 on smithi111 2024-04-07T14:03:24.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:24 smithi111 ceph-mon[24802]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:24.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:24 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:24.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:24 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.rtrlso", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:03:24.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:24 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:03:24.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:24 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:24 smithi145 ceph-mon[26800]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:24 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:24 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi111.rtrlso", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:03:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:24 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:03:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:24 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:25.648 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:25 smithi111 ceph-mon[24802]: Reconfiguring mgr.smithi111.rtrlso (unknown last config time)... 2024-04-07T14:03:25.649 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:25 smithi111 ceph-mon[24802]: Reconfiguring daemon mgr.smithi111.rtrlso on smithi111 2024-04-07T14:03:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:25 smithi145 ceph-mon[26800]: Reconfiguring mgr.smithi111.rtrlso (unknown last config time)... 2024-04-07T14:03:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:25 smithi145 ceph-mon[26800]: Reconfiguring daemon mgr.smithi111.rtrlso on smithi111 2024-04-07T14:03:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:26 smithi111 ceph-mon[24802]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:26 smithi111 ceph-mon[24802]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-04-07T14:03:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:26 smithi111 ceph-mon[24802]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-04-07T14:03:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:26 smithi145 ceph-mon[26800]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:26 smithi145 ceph-mon[26800]: Reconfiguring alertmanager.smithi111 (dependencies changed)... 2024-04-07T14:03:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:26 smithi145 ceph-mon[26800]: Reconfiguring daemon alertmanager.smithi111 on smithi111 2024-04-07T14:03:27.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:27 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-07T14:03:27.885 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:27 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:28.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:27 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-07T14:03:28.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:27 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:28.791 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:28 smithi111 ceph-mon[24802]: Deploying daemon osd.0 on smithi111 2024-04-07T14:03:28.791 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:28 smithi111 ceph-mon[24802]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:28 smithi145 ceph-mon[26800]: Deploying daemon osd.0 on smithi111 2024-04-07T14:03:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:28 smithi145 ceph-mon[26800]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:30.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:30 smithi111 ceph-mon[24802]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:31.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:30 smithi145 ceph-mon[26800]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:32 smithi111 ceph-mon[24802]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:03:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:33.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:32 smithi145 ceph-mon[26800]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi111", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:03:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:33 smithi111 ceph-mon[24802]: Reconfiguring crash.smithi111 (monmap changed)... 2024-04-07T14:03:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:33 smithi111 ceph-mon[24802]: Reconfiguring daemon crash.smithi111 on smithi111 2024-04-07T14:03:34.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:33 smithi145 ceph-mon[26800]: Reconfiguring crash.smithi111 (monmap changed)... 2024-04-07T14:03:34.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:33 smithi145 ceph-mon[26800]: Reconfiguring daemon crash.smithi111 on smithi111 2024-04-07T14:03:34.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:34 smithi111 ceph-mon[24802]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:35.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:34 smithi145 ceph-mon[26800]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:36.406 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:36 smithi111 ceph-mon[24802]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:36.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:36 smithi145 ceph-mon[26800]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:38.258 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:38.258 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:38 smithi111 ceph-mon[24802]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:38.258 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:03:38.259 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:03:38.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:38.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:38 smithi145 ceph-mon[26800]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:03:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:03:39.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:39 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:39.421 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:39 smithi111 ceph-mon[24802]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-04-07T14:03:39.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:39 smithi111 ceph-mon[24802]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-04-07T14:03:39.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:39 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:39.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:39 smithi145 ceph-mon[26800]: Reconfiguring grafana.smithi111 (dependencies changed)... 2024-04-07T14:03:39.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:39 smithi145 ceph-mon[26800]: Reconfiguring daemon grafana.smithi111 on smithi111 2024-04-07T14:03:40.354 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:40 smithi111 ceph-mon[24802]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:40.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:40 smithi145 ceph-mon[26800]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:41.384 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:41 smithi111 ceph-mon[24802]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T14:03:41.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:41 smithi145 ceph-mon[26800]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-07T14:03:42.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:42 smithi111 ceph-mon[24802]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-07T14:03:42.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:42 smithi111 ceph-mon[24802]: osdmap e7: 1 total, 0 up, 1 in 2024-04-07T14:03:42.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:42.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:42 smithi111 ceph-mon[24802]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:03:42.429 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:42 smithi111 ceph-mon[24802]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:42.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:42 smithi145 ceph-mon[26800]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-07T14:03:42.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:42 smithi145 ceph-mon[26800]: osdmap e7: 1 total, 0 up, 1 in 2024-04-07T14:03:42.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:42.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:42 smithi145 ceph-mon[26800]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:03:42.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:42 smithi145 ceph-mon[26800]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: osdmap e8: 1 total, 0 up, 1 in 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: Reconfiguring crash.smithi145 (monmap changed)... 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:43.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:43 smithi111 ceph-mon[24802]: Reconfiguring daemon crash.smithi145 on smithi145 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: osdmap e8: 1 total, 0 up, 1 in 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:43.364 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: Reconfiguring crash.smithi145 (monmap changed)... 2024-04-07T14:03:43.365 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi145", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-04-07T14:03:43.365 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:43.365 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:43 smithi145 ceph-mon[26800]: Reconfiguring daemon crash.smithi145 on smithi145 2024-04-07T14:03:43.924 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 0 on host 'smithi111' 2024-04-07T14:03:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:03:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:03:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062] boot 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: osdmap e9: 1 total, 1 up, 1 in 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:03:44.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:44 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: osd.0 [v2:172.21.15.111:6802/2388610062,v1:172.21.15.111:6803/2388610062] boot 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: osdmap e9: 1 total, 1 up, 1 in 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:03:44.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-07T14:03:44.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:44.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:44.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:03:44.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:03:44.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:44 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:44.515 DEBUG:teuthology.orchestra.run.smithi111:osd.0> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.0.service 2024-04-07T14:03:44.518 INFO:tasks.cephadm:Deploying osd.1 on smithi111 with /dev/vg_nvme/lv_3... 2024-04-07T14:03:44.518 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-07T14:03:45.390 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:45 smithi145 ceph-mon[26800]: Reconfiguring mgr.smithi145.qiunpb (monmap changed)... 2024-04-07T14:03:45.390 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:45 smithi145 ceph-mon[26800]: Reconfiguring daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:03:45.390 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:45 smithi145 ceph-mon[26800]: osdmap e10: 1 total, 1 up, 1 in 2024-04-07T14:03:45.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:45 smithi111 ceph-mon[24802]: Reconfiguring mgr.smithi145.qiunpb (monmap changed)... 2024-04-07T14:03:45.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:45 smithi111 ceph-mon[24802]: Reconfiguring daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:03:45.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:45 smithi111 ceph-mon[24802]: osdmap e10: 1 total, 1 up, 1 in 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112176 s, 935 MB/s 2024-04-07T14:03:46.502 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-07T14:03:46.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:46.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: Reconfiguring mon.smithi145 (monmap changed)... 2024-04-07T14:03:46.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:03:46.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T14:03:46.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:46.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: Reconfiguring daemon mon.smithi145 on smithi145 2024-04-07T14:03:46.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:46 smithi145 ceph-mon[26800]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: Reconfiguring mon.smithi145 (monmap changed)... 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: Reconfiguring daemon mon.smithi145 on smithi145 2024-04-07T14:03:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:46 smithi111 ceph-mon[24802]: pgmap v32: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:47.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi111:vg_nvme/lv_3 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.111:9093"}]: dispatch 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.111:9093"}]: dispatch 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:47.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.111:9093"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://172.21.15.111:9093"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-04-07T14:03:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://172.21.15.111:3000"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:48.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:03:48.971 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:48 smithi145 ceph-mon[26800]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-04-07T14:03:48.971 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:48 smithi145 ceph-mon[26800]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:48.971 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:48.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:48 smithi111 ceph-mon[24802]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://172.21.15.111:9095"}]: dispatch 2024-04-07T14:03:48.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:48 smithi111 ceph-mon[24802]: pgmap v33: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:48.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:50 smithi111 ceph-mon[24802]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:50 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:03:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:50 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:03:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:50 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:50 smithi145 ceph-mon[26800]: pgmap v34: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:50 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:03:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:50 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:03:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:50 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:03:51.936 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:51 smithi111 ceph-mon[24802]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:03:52.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:51 smithi145 ceph-mon[26800]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:03:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:52 smithi111 ceph-mon[24802]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:53.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:52 smithi145 ceph-mon[26800]: pgmap v35: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:54.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:53 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:54.092 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:53 smithi111 ceph-mon[24802]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:54.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:53 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:03:54.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:53 smithi145 ceph-mon[26800]: pgmap v36: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:54.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:54 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3226062210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0ebc0e4-c2a4-49ef-bf0c-43bc03873463"}]: dispatch 2024-04-07T14:03:54.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:54 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3226062210' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0ebc0e4-c2a4-49ef-bf0c-43bc03873463"}]': finished 2024-04-07T14:03:54.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:54 smithi111 ceph-mon[24802]: osdmap e11: 2 total, 1 up, 2 in 2024-04-07T14:03:54.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:54 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:03:55.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3226062210' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "d0ebc0e4-c2a4-49ef-bf0c-43bc03873463"}]: dispatch 2024-04-07T14:03:55.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3226062210' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "d0ebc0e4-c2a4-49ef-bf0c-43bc03873463"}]': finished 2024-04-07T14:03:55.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:54 smithi145 ceph-mon[26800]: osdmap e11: 2 total, 1 up, 2 in 2024-04-07T14:03:55.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:54 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:03:56.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:55 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/4094520391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:03:56.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:55 smithi111 ceph-mon[24802]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:56.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:55 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/4094520391' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:03:56.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:55 smithi145 ceph-mon[26800]: pgmap v38: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:58.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:03:58 smithi111 ceph-mon[24802]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:03:58.976 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:03:58 smithi145 ceph-mon[26800]: pgmap v39: 0 pgs: ; 0 B data, 4.8 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi111 2024-04-07T14:04:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi111 to 12864M 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:00.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: pgmap v40: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi111 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi111 to 12864M 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:00.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-07T14:04:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:02 smithi111 ceph-mon[24802]: Deploying daemon osd.1 on smithi111 2024-04-07T14:04:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:02 smithi111 ceph-mon[24802]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-07T14:04:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:02 smithi145 ceph-mon[26800]: Deploying daemon osd.1 on smithi111 2024-04-07T14:04:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:02 smithi145 ceph-mon[26800]: pgmap v41: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:04.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:04 smithi111 ceph-mon[24802]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:05.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:04 smithi145 ceph-mon[26800]: pgmap v42: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:06.353 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:06 smithi111 ceph-mon[24802]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:06.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:06 smithi145 ceph-mon[26800]: pgmap v43: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:07.868 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:07.868 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:08.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:08.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:08 smithi111 ceph-mon[24802]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:08 smithi145 ceph-mon[26800]: pgmap v44: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:10.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:10 smithi111 ceph-mon[24802]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:10.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:11.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:10 smithi145 ceph-mon[26800]: pgmap v45: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:11.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:11.690 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:11 smithi111 ceph-mon[24802]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T14:04:11.724 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 1 on host 'smithi111' 2024-04-07T14:04:12.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:11 smithi145 ceph-mon[26800]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-07T14:04:12.426 DEBUG:teuthology.orchestra.run.smithi111:osd.1> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.1.service 2024-04-07T14:04:12.428 INFO:tasks.cephadm:Deploying osd.2 on smithi111 with /dev/vg_nvme/lv_2... 2024-04-07T14:04:12.428 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: osdmap e12: 2 total, 1 up, 2 in 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:04:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:12.727 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1[39298]: 2024-04-07T14:04:12.630+0000 7f51b79ed700 -1 osd.1 0 waiting for initial osdmap 2024-04-07T14:04:12.727 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:12 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1[39298]: 2024-04-07T14:04:12.633+0000 7f51b4388700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: pgmap v46: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: osdmap e12: 2 total, 1 up, 2 in 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:04:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:13.869 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:13 smithi111 ceph-mon[24802]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:04:13.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:13 smithi111 ceph-mon[24802]: osdmap e13: 2 total, 1 up, 2 in 2024-04-07T14:04:13.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:13 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:13.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:13 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:13.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:13 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:14.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:13 smithi145 ceph-mon[26800]: from='osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:04:14.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:13 smithi145 ceph-mon[26800]: osdmap e13: 2 total, 1 up, 2 in 2024-04-07T14:04:14.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:13 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:14.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:13 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:14.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:13 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130018 s, 806 MB/s 2024-04-07T14:04:14.610 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298] boot 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: osdmap e14: 2 total, 2 up, 2 in 2024-04-07T14:04:14.837 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:15.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:04:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:04:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: pgmap v49: 0 pgs: ; 0 B data, 4.9 MiB used, 89 GiB / 89 GiB avail 2024-04-07T14:04:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: osd.1 [v2:172.21.15.111:6810/1959509298,v1:172.21.15.111:6811/1959509298] boot 2024-04-07T14:04:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: osdmap e14: 2 total, 2 up, 2 in 2024-04-07T14:04:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:04:15.531 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi111:vg_nvme/lv_2 2024-04-07T14:04:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:15 smithi145 ceph-mon[26800]: osdmap e15: 2 total, 2 up, 2 in 2024-04-07T14:04:15.860 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:15 smithi111 ceph-mon[24802]: osdmap e15: 2 total, 2 up, 2 in 2024-04-07T14:04:16.839 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:16 smithi111 ceph-mon[24802]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:17.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:16 smithi145 ceph-mon[26800]: pgmap v52: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:18.832 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:18 smithi111 ceph-mon[24802]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:18.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:04:18.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:04:18.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:18.833 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:18 smithi145 ceph-mon[26800]: pgmap v53: 0 pgs: ; 0 B data, 9.7 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:04:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:04:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:19.691 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:19 smithi111 ceph-mon[24802]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:04:20.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:19 smithi145 ceph-mon[26800]: from='client.14268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:04:20.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:20 smithi111 ceph-mon[24802]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:20.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:20 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/75666393' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed8d6081-0935-4cab-987b-7393b5e15e12"}]: dispatch 2024-04-07T14:04:20.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:20 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/75666393' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed8d6081-0935-4cab-987b-7393b5e15e12"}]': finished 2024-04-07T14:04:20.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:20 smithi111 ceph-mon[24802]: osdmap e16: 3 total, 2 up, 3 in 2024-04-07T14:04:20.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:20 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:20 smithi145 ceph-mon[26800]: pgmap v54: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:20 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/75666393' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ed8d6081-0935-4cab-987b-7393b5e15e12"}]: dispatch 2024-04-07T14:04:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:20 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/75666393' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ed8d6081-0935-4cab-987b-7393b5e15e12"}]': finished 2024-04-07T14:04:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:20 smithi145 ceph-mon[26800]: osdmap e16: 3 total, 2 up, 3 in 2024-04-07T14:04:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:20 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:21.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:21 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1192878251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:04:22.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:21 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1192878251' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:04:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:22 smithi111 ceph-mon[24802]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:22 smithi145 ceph-mon[26800]: pgmap v56: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:24.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:24 smithi111 ceph-mon[24802]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:24 smithi145 ceph-mon[26800]: pgmap v57: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi111 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi111 to 6432M 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:26.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:26.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:26.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: pgmap v58: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi111 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi111 to 6432M 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:27.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:28.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:28 smithi111 ceph-mon[24802]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:28.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:28 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-07T14:04:28.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:28 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:28.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:28 smithi145 ceph-mon[26800]: pgmap v59: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:28.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:28 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-07T14:04:28.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:28 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:29.835 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:29 smithi111 ceph-mon[24802]: Deploying daemon osd.2 on smithi111 2024-04-07T14:04:29.837 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:29 smithi145 ceph-mon[26800]: Deploying daemon osd.2 on smithi111 2024-04-07T14:04:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:30 smithi111 ceph-mon[24802]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:31.004 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:30 smithi145 ceph-mon[26800]: pgmap v60: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:32.968 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:32 smithi111 ceph-mon[24802]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:32 smithi145 ceph-mon[26800]: pgmap v61: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:34.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:34.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:34 smithi111 ceph-mon[24802]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:35.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:34 smithi145 ceph-mon[26800]: pgmap v62: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:36.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:36.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:36.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:36.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:36.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:35 smithi111 ceph-mon[24802]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:36.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:36.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:36.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:36.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:36.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:35 smithi145 ceph-mon[26800]: pgmap v63: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:37.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:04:37.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:04:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:04:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:04:38.419 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 2 on host 'smithi111' 2024-04-07T14:04:38.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:38 smithi111 ceph-mon[24802]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:38.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:38 smithi111 ceph-mon[24802]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T14:04:38.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:38 smithi145 ceph-mon[26800]: pgmap v64: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:38 smithi145 ceph-mon[26800]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-07T14:04:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:39.401 DEBUG:teuthology.orchestra.run.smithi111:osd.2> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.2.service 2024-04-07T14:04:39.404 INFO:tasks.cephadm:Deploying osd.3 on smithi111 with /dev/vg_nvme/lv_1... 2024-04-07T14:04:39.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: osdmap e17: 3 total, 2 up, 3 in 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: osdmap e18: 3 total, 2 up, 3 in 2024-04-07T14:04:39.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:39.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:39.977 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2[43000]: 2024-04-07T14:04:39.642+0000 7fb331940700 -1 osd.2 0 waiting for initial osdmap 2024-04-07T14:04:39.977 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:39 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2[43000]: 2024-04-07T14:04:39.646+0000 7fb32c2d7700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: osdmap e17: 3 total, 2 up, 3 in 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: osdmap e18: 3 total, 2 up, 3 in 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:40.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:39 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:40.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:40 smithi145 ceph-mon[26800]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:40.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:40 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:40.955 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:40 smithi111 ceph-mon[24802]: pgmap v66: 0 pgs: ; 0 B data, 9.8 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:40.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:41.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:41 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:04:41.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:41 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:04:41.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:41 smithi111 ceph-mon[24802]: osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539] boot 2024-04-07T14:04:41.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:41 smithi111 ceph-mon[24802]: osdmap e19: 3 total, 3 up, 3 in 2024-04-07T14:04:41.938 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:41 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:42.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:41 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:04:42.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:41 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:04:42.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:41 smithi145 ceph-mon[26800]: osd.2 [v2:172.21.15.111:6818/2995507539,v1:172.21.15.111:6819/2995507539] boot 2024-04-07T14:04:42.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:41 smithi145 ceph-mon[26800]: osdmap e19: 3 total, 3 up, 3 in 2024-04-07T14:04:42.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:41 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10+0 records in 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:10+0 records out 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0119091 s, 880 MB/s 2024-04-07T14:04:42.921 INFO:teuthology.orchestra.run.smithi111.stderr:--> Zapping successful for: 2024-04-07T14:04:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:42 smithi111 ceph-mon[24802]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-07T14:04:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-07T14:04:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:42 smithi111 ceph-mon[24802]: osdmap e20: 3 total, 3 up, 3 in 2024-04-07T14:04:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-07T14:04:43.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:42 smithi145 ceph-mon[26800]: pgmap v69: 0 pgs: ; 0 B data, 9.9 MiB used, 179 GiB / 179 GiB avail 2024-04-07T14:04:43.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-04-07T14:04:43.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-04-07T14:04:43.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:42 smithi145 ceph-mon[26800]: osdmap e20: 3 total, 3 up, 3 in 2024-04-07T14:04:43.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-07T14:04:43.535 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi111:vg_nvme/lv_1 2024-04-07T14:04:43.777 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44426]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T14:04:43.777 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44426]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:43.778 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44426]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:44.051 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44460]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T14:04:44.051 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44460]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:44.051 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:04:43 smithi111 sudo[44460]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:44.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-07T14:04:44.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:43 smithi111 ceph-mon[24802]: osdmap e21: 3 total, 3 up, 3 in 2024-04-07T14:04:44.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:44.051 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:43 smithi111 ceph-mon[24802]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:44.052 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-07T14:04:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:43 smithi145 ceph-mon[26800]: osdmap e21: 3 total, 3 up, 3 in 2024-04-07T14:04:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:43 smithi145 ceph-mon[26800]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:44.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:44.307 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44494]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T14:04:44.308 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44494]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:44.308 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44494]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:44.558 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44497]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T14:04:44.558 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44497]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:44.558 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44497]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:44.815 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44518]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T14:04:44.815 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44518]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:44.815 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44518]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:45.076 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44529]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T14:04:45.076 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44529]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:45.076 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44529]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:45.076 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44521]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T14:04:45.076 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44521]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:45.076 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:04:44 smithi111 sudo[44521]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:45.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44551]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T14:04:45.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44551]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:45.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 sudo[44551]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:45.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29125]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-07T14:04:45.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29125]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:45.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29125]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:46.114 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:45 smithi111 ceph-mon[24802]: osdmap e22: 3 total, 3 up, 3 in 2024-04-07T14:04:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29128]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-07T14:04:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29128]: pam_unix(sudo:session): session opened for user root by (uid=0) 2024-04-07T14:04:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 sudo[29128]: pam_unix(sudo:session): session closed for user root 2024-04-07T14:04:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:45 smithi145 ceph-mon[26800]: osdmap e22: 3 total, 3 up, 3 in 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: mgrmap e19: smithi111.rtrlso(active, since 2m), standbys: smithi145.qiunpb 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:04:47.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:04:47.081 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:46 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: mgrmap e19: smithi111.rtrlso(active, since 2m), standbys: smithi145.qiunpb 2024-04-07T14:04:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 16 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='client.14292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi111:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:04:47.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:46 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:48.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:47 smithi111 ceph-mon[24802]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:48.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:47 smithi145 ceph-mon[26800]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 17 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:49.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi111 to 4288M 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:49.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:48 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi111 to 4288M 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:04:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:48 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1750351723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "febc59b7-eb37-4e44-9339-46400b69a401"}]: dispatch 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1750351723' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "febc59b7-eb37-4e44-9339-46400b69a401"}]': finished 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: osdmap e23: 4 total, 3 up, 4 in 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:49 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3383960739' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:04:50.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1750351723' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "febc59b7-eb37-4e44-9339-46400b69a401"}]: dispatch 2024-04-07T14:04:50.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1750351723' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "febc59b7-eb37-4e44-9339-46400b69a401"}]': finished 2024-04-07T14:04:50.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: osdmap e23: 4 total, 3 up, 4 in 2024-04-07T14:04:50.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:04:50.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:04:50.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:49 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3383960739' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:04:52.664 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:52 smithi111 ceph-mon[24802]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-07T14:04:53.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:52 smithi145 ceph-mon[26800]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 19 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2024-04-07T14:04:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:54 smithi111 ceph-mon[24802]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-07T14:04:55.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:54 smithi145 ceph-mon[26800]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2024-04-07T14:04:56.684 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:56 smithi111 ceph-mon[24802]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T14:04:56.684 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:56 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-07T14:04:56.684 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:56 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:56.684 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:56 smithi111 ceph-mon[24802]: Deploying daemon osd.3 on smithi111 2024-04-07T14:04:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:56 smithi145 ceph-mon[26800]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T14:04:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:56 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-07T14:04:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:56 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:04:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:56 smithi145 ceph-mon[26800]: Deploying daemon osd.3 on smithi111 2024-04-07T14:04:58.860 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:04:58 smithi111 ceph-mon[24802]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T14:04:59.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:04:58 smithi145 ceph-mon[26800]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.8 KiB/s rd, 78 KiB/s wr, 5 op/s 2024-04-07T14:05:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:00 smithi111 ceph-mon[24802]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-04-07T14:05:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:00.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:00 smithi145 ceph-mon[26800]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.7 KiB/s rd, 76 KiB/s wr, 5 op/s 2024-04-07T14:05:00.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:00.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:00.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:00.825 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:02.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:02 smithi111 ceph-mon[24802]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-07T14:05:03.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:02 smithi145 ceph-mon[26800]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail; 1.5 KiB/s rd, 65 KiB/s wr, 4 op/s 2024-04-07T14:05:03.361 INFO:teuthology.orchestra.run.smithi111.stdout:Created osd(s) 3 on host 'smithi111' 2024-04-07T14:05:04.013 DEBUG:teuthology.orchestra.run.smithi111:osd.3> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.3.service 2024-04-07T14:05:04.015 INFO:tasks.cephadm:Deploying osd.4 on smithi145 with /dev/vg_nvme/lv_4... 2024-04-07T14:05:04.016 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-07T14:05:04.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:04 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:04.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:04 smithi111 ceph-mon[24802]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:05:04.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:04 smithi111 ceph-mon[24802]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T14:05:04.583 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:04 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:04.583 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:04 smithi145 ceph-mon[26800]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:05:04.583 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:04 smithi145 ceph-mon[26800]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-07T14:05:05.503 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:05 smithi145 ceph-mon[26800]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-07T14:05:05.503 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:05 smithi145 ceph-mon[26800]: osdmap e24: 4 total, 3 up, 4 in 2024-04-07T14:05:05.503 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:05.503 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:05 smithi145 ceph-mon[26800]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:05:05.712 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-mon[24802]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-07T14:05:05.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-mon[24802]: osdmap e24: 4 total, 3 up, 4 in 2024-04-07T14:05:05.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:05.713 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-mon[24802]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]: dispatch 2024-04-07T14:05:05.713 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3[46261]: 2024-04-07T14:05:05.373+0000 7f44475a5700 -1 osd.3 0 waiting for initial osdmap 2024-04-07T14:05:05.713 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:05:05 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3[46261]: 2024-04-07T14:05:05.378+0000 7f4442f3e700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112165 s, 935 MB/s 2024-04-07T14:05:06.025 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-04-07T14:05:06.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:06 smithi145 ceph-mon[26800]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:05:06.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:06 smithi145 ceph-mon[26800]: osdmap e25: 4 total, 3 up, 4 in 2024-04-07T14:05:06.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:06 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:06.660 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:06 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:06.661 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:06 smithi145 ceph-mon[26800]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:05:06.682 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi145:vg_nvme/lv_4 2024-04-07T14:05:06.725 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:06 smithi111 ceph-mon[24802]: from='osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi111", "root=default"]}]': finished 2024-04-07T14:05:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:06 smithi111 ceph-mon[24802]: osdmap e25: 4 total, 3 up, 4 in 2024-04-07T14:05:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:06 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:06 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:06 smithi111 ceph-mon[24802]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 18 MiB used, 268 GiB / 268 GiB avail 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577] boot 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: osdmap e26: 4 total, 4 up, 4 in 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:07.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: osd.3 [v2:172.21.15.111:6826/1526297577,v1:172.21.15.111:6827/1526297577] boot 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: osdmap e26: 4 total, 4 up, 4 in 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:05:07.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:08.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:08 smithi145 ceph-mon[26800]: osdmap e27: 4 total, 4 up, 4 in 2024-04-07T14:05:08.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:08 smithi145 ceph-mon[26800]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:08.690 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:08 smithi111 ceph-mon[24802]: osdmap e27: 4 total, 4 up, 4 in 2024-04-07T14:05:08.690 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:08 smithi111 ceph-mon[24802]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:09 smithi111 ceph-mon[24802]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:05:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:09 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:09 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:09.640 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:09 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:09.724 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:09 smithi145 ceph-mon[26800]: from='client.24137 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:05:09.725 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:09 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:09.725 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:09 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:09.725 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:09 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:10.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:10.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:10.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:10.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:10.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:10.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi111 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi111 to 3216M 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/16190473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]: dispatch 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]: dispatch 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]': finished 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: osdmap e28: 5 total, 4 up, 5 in 2024-04-07T14:05:11.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:11 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi111 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi111 to 3216M 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/16190473' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]: dispatch 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]: dispatch 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce"}]': finished 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: osdmap e28: 5 total, 4 up, 5 in 2024-04-07T14:05:11.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:11 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:12 smithi111 ceph-mon[24802]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:12.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:12 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/4087559220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:05:12.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:12 smithi145 ceph-mon[26800]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:12.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:12 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/4087559220' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:05:14.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:14 smithi145 ceph-mon[26800]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:14 smithi111 ceph-mon[24802]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:16.526 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:16 smithi145 ceph-mon[26800]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:16.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:16 smithi111 ceph-mon[24802]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:17.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-07T14:05:17.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:17.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-07T14:05:17.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:18.568 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:18 smithi145 ceph-mon[26800]: Deploying daemon osd.4 on smithi145 2024-04-07T14:05:18.569 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:18 smithi145 ceph-mon[26800]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:18.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:18 smithi111 ceph-mon[24802]: Deploying daemon osd.4 on smithi145 2024-04-07T14:05:18.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:18 smithi111 ceph-mon[24802]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:20.879 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:20 smithi111 ceph-mon[24802]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:20 smithi145 ceph-mon[26800]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:22.865 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:22 smithi145 ceph-mon[26800]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:22.865 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:22.865 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:22.865 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:22.865 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:22 smithi111 ceph-mon[24802]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:24.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:24 smithi145 ceph-mon[26800]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:24.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:24 smithi111 ceph-mon[24802]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:25.159 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 4 on host 'smithi145' 2024-04-07T14:05:25.863 DEBUG:teuthology.orchestra.run.smithi145:osd.4> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.4.service 2024-04-07T14:05:25.866 INFO:tasks.cephadm:Deploying osd.5 on smithi145 with /dev/vg_nvme/lv_3... 2024-04-07T14:05:25.866 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-07T14:05:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:26 smithi111 ceph-mon[24802]: from='osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T14:05:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:26 smithi111 ceph-mon[24802]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T14:05:26.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:26 smithi111 ceph-mon[24802]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:26.487 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:26.487 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:26 smithi145 ceph-mon[26800]: from='osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T14:05:26.487 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:26 smithi145 ceph-mon[26800]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-07T14:05:26.487 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:26 smithi145 ceph-mon[26800]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:27.443 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-mon[26800]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-07T14:05:27.443 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-mon[26800]: osdmap e29: 5 total, 4 up, 5 in 2024-04-07T14:05:27.443 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:27.443 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-mon[26800]: from='osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:27.443 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-mon[26800]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:27.443 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:05:27.353+0000 7f998082e700 -1 osd.4 0 waiting for initial osdmap 2024-04-07T14:05:27.443 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:05:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:05:27.357+0000 7f997c9c8700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:05:27.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:27 smithi111 ceph-mon[24802]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-07T14:05:27.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:27 smithi111 ceph-mon[24802]: osdmap e29: 5 total, 4 up, 5 in 2024-04-07T14:05:27.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:27 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:27.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:27 smithi111 ceph-mon[24802]: from='osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:27.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:27 smithi111 ceph-mon[24802]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:27.903 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-07T14:05:27.903 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-07T14:05:27.904 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-04-07T14:05:27.904 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-04-07T14:05:27.904 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123032 s, 852 MB/s 2024-04-07T14:05:27.904 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-04-07T14:05:28.631 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:28 smithi145 ceph-mon[26800]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:05:28.631 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:28 smithi145 ceph-mon[26800]: osdmap e30: 5 total, 4 up, 5 in 2024-04-07T14:05:28.631 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:28 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:28.632 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:28 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:28.632 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:28 smithi145 ceph-mon[26800]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:28 smithi111 ceph-mon[24802]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:05:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:28 smithi111 ceph-mon[24802]: osdmap e30: 5 total, 4 up, 5 in 2024-04-07T14:05:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:28 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:28 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:28 smithi111 ceph-mon[24802]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 23 MiB used, 358 GiB / 358 GiB avail 2024-04-07T14:05:29.018 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi145:vg_nvme/lv_3 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651] boot 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: osdmap e31: 5 total, 5 up, 5 in 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:29.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: osd.4 [v2:172.21.15.145:6800/3801945651,v1:172.21.15.145:6801/3801945651] boot 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: osdmap e31: 5 total, 5 up, 5 in 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:05:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:30 smithi111 ceph-mon[24802]: osdmap e32: 5 total, 5 up, 5 in 2024-04-07T14:05:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:30 smithi111 ceph-mon[24802]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:30 smithi111 ceph-mon[24802]: osdmap e33: 5 total, 5 up, 5 in 2024-04-07T14:05:30.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:30 smithi145 ceph-mon[26800]: osdmap e32: 5 total, 5 up, 5 in 2024-04-07T14:05:30.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:30 smithi145 ceph-mon[26800]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:30.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:30 smithi145 ceph-mon[26800]: osdmap e33: 5 total, 5 up, 5 in 2024-04-07T14:05:32.679 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:32 smithi145 ceph-mon[26800]: osdmap e34: 5 total, 5 up, 5 in 2024-04-07T14:05:32.679 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:32 smithi145 ceph-mon[26800]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:32 smithi111 ceph-mon[24802]: osdmap e34: 5 total, 5 up, 5 in 2024-04-07T14:05:32.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:32 smithi111 ceph-mon[24802]: pgmap v109: 1 pgs: 1 remapped+peering; 577 KiB data, 28 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: osdmap e35: 5 total, 5 up, 5 in 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:33.385 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:33.386 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: osdmap e35: 5 total, 5 up, 5 in 2024-04-07T14:05:33.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='client.24155 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:05:33.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:33.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:34.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:34 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi145 2024-04-07T14:05:34.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:34 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi145 to 15936M 2024-04-07T14:05:34.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:34 smithi145 ceph-mon[26800]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:34 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi145 2024-04-07T14:05:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:34 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi145 to 15936M 2024-04-07T14:05:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:34 smithi111 ceph-mon[24802]: pgmap v111: 1 pgs: 1 remapped+peering; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:35.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:35 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/2895236181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]: dispatch 2024-04-07T14:05:35.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:35 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]: dispatch 2024-04-07T14:05:35.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:35 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]': finished 2024-04-07T14:05:35.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:35 smithi111 ceph-mon[24802]: osdmap e36: 6 total, 5 up, 6 in 2024-04-07T14:05:35.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:35 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/2895236181' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]: dispatch 2024-04-07T14:05:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:35 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]: dispatch 2024-04-07T14:05:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:35 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4fc8f366-9ee3-4b85-961d-db120af9f964"}]': finished 2024-04-07T14:05:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:35 smithi145 ceph-mon[26800]: osdmap e36: 6 total, 5 up, 6 in 2024-04-07T14:05:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:36.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:36 smithi111 ceph-mon[24802]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-07T14:05:36.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:36 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/886374232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:05:36.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:36 smithi145 ceph-mon[26800]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 100 KiB/s, 0 objects/s recovering 2024-04-07T14:05:36.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:36 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/886374232' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:05:37.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:05:37.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:05:38.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:05:38.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:05:38.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:38 smithi145 ceph-mon[26800]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-07T14:05:38.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:38 smithi111 ceph-mon[24802]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-04-07T14:05:40.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:40 smithi111 ceph-mon[24802]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T14:05:40.984 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:40 smithi145 ceph-mon[26800]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T14:05:41.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:41 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-07T14:05:41.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:41 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:42.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:41 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-07T14:05:42.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:41 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:42.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:42 smithi111 ceph-mon[24802]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T14:05:42.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:42 smithi111 ceph-mon[24802]: Deploying daemon osd.5 on smithi145 2024-04-07T14:05:43.004 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:42 smithi145 ceph-mon[26800]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T14:05:43.004 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:42 smithi145 ceph-mon[26800]: Deploying daemon osd.5 on smithi145 2024-04-07T14:05:44.966 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:44 smithi145 ceph-mon[26800]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:05:44.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:44 smithi111 ceph-mon[24802]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:05:46.521 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:46 smithi145 ceph-mon[26800]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:46.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:46 smithi111 ceph-mon[24802]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:47 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:47.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:47 smithi111 ceph-mon[24802]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:47 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:48.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:47 smithi145 ceph-mon[26800]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:49.754 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 5 on host 'smithi145' 2024-04-07T14:05:50.742 DEBUG:teuthology.orchestra.run.smithi145:osd.5> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.5.service 2024-04-07T14:05:50.744 INFO:tasks.cephadm:Deploying osd.6 on smithi145 with /dev/vg_nvme/lv_2... 2024-04-07T14:05:50.745 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-07T14:05:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:50 smithi145 ceph-mon[26800]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:50 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:50 smithi145 ceph-mon[26800]: from='osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T14:05:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:50 smithi145 ceph-mon[26800]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T14:05:51.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:50 smithi111 ceph-mon[24802]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:51.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:50 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:51.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:50 smithi111 ceph-mon[24802]: from='osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T14:05:51.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:50 smithi111 ceph-mon[24802]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-07T14:05:51.896 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-mon[26800]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-07T14:05:51.896 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-mon[26800]: osdmap e37: 6 total, 5 up, 6 in 2024-04-07T14:05:51.897 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:51.897 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-mon[26800]: from='osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:51.897 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-mon[26800]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:51.897 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:05:51.770+0000 7fbdf25c0700 -1 osd.5 0 waiting for initial osdmap 2024-04-07T14:05:51.897 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:05:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:05:51.775+0000 7fbdec756700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:05:52.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:51 smithi111 ceph-mon[24802]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-07T14:05:52.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:51 smithi111 ceph-mon[24802]: osdmap e37: 6 total, 5 up, 6 in 2024-04-07T14:05:52.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:51 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:51 smithi111 ceph-mon[24802]: from='osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:52.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:51 smithi111 ceph-mon[24802]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:05:52.966 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: osdmap e38: 6 total, 5 up, 6 in 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709] boot 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: osdmap e39: 6 total, 6 up, 6 in 2024-04-07T14:05:52.967 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:52 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 29 MiB used, 447 GiB / 447 GiB avail 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: osdmap e38: 6 total, 5 up, 6 in 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: osd.5 [v2:172.21.15.145:6808/2218575709,v1:172.21.15.145:6809/2218575709] boot 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: osdmap e39: 6 total, 6 up, 6 in 2024-04-07T14:05:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:52 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:05:53.989 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:53 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:05:53.990 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:53 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:05:53.990 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:53 smithi145 ceph-mon[26800]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:53.990 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:53 smithi145 ceph-mon[26800]: osdmap e40: 6 total, 6 up, 6 in 2024-04-07T14:05:54.106 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-07T14:05:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-07T14:05:54.107 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-04-07T14:05:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-04-07T14:05:54.107 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128578 s, 816 MB/s 2024-04-07T14:05:54.107 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-04-07T14:05:54.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:53 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:05:54.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:53 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:05:54.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:53 smithi111 ceph-mon[24802]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:54.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:53 smithi111 ceph-mon[24802]: osdmap e40: 6 total, 6 up, 6 in 2024-04-07T14:05:55.398 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi145:vg_nvme/lv_2 2024-04-07T14:05:56.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:55 smithi145 ceph-mon[26800]: osdmap e41: 6 total, 6 up, 6 in 2024-04-07T14:05:56.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:55 smithi145 ceph-mon[26800]: osdmap e42: 6 total, 6 up, 6 in 2024-04-07T14:05:56.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:55 smithi145 ceph-mon[26800]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:56.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:55 smithi111 ceph-mon[24802]: osdmap e41: 6 total, 6 up, 6 in 2024-04-07T14:05:56.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:55 smithi111 ceph-mon[24802]: osdmap e42: 6 total, 6 up, 6 in 2024-04-07T14:05:56.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:55 smithi111 ceph-mon[24802]: pgmap v129: 1 pgs: 1 remapped+peering; 577 KiB data, 34 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi145 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi145 to 7968M 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:57.367 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:57.368 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.368 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: osdmap e43: 6 total, 6 up, 6 in 2024-04-07T14:05:57.368 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:57 smithi145 ceph-mon[26800]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi145 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi145 to 7968M 2024-04-07T14:05:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: osdmap e43: 6 total, 6 up, 6 in 2024-04-07T14:05:57.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:57 smithi111 ceph-mon[24802]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T14:05:58.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:58 smithi111 ceph-mon[24802]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:58.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:58 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:58.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:58 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:58.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:58 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:58.522 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:58 smithi145 ceph-mon[26800]: pgmap v131: 1 pgs: 1 remapped+peering; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:05:58.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:58 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:05:58.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:58 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:05:58.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:58 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:05:59.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:05:59 smithi111 ceph-mon[24802]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:05:59.509 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:05:59 smithi145 ceph-mon[26800]: from='client.24173 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:06:00.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:00 smithi111 ceph-mon[24802]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:00.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:00 smithi145 ceph-mon[26800]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: Cluster is now healthy 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/129846119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]: dispatch 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]: dispatch 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]': finished 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: osdmap e44: 7 total, 6 up, 7 in 2024-04-07T14:06:01.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:01.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:01 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/1344333657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: Cluster is now healthy 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/129846119' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]: dispatch 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]: dispatch 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "5a86e7a8-01c0-4188-9774-cc71681e8664"}]': finished 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: osdmap e44: 7 total, 6 up, 7 in 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:01.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:01 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/1344333657' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:06:02.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:02 smithi111 ceph-mon[24802]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-07T14:06:02.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:02 smithi145 ceph-mon[26800]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-04-07T14:06:04.655 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:04 smithi145 ceph-mon[26800]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T14:06:04.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:04 smithi111 ceph-mon[24802]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-07T14:06:06.674 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:06 smithi145 ceph-mon[26800]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T14:06:06.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:06 smithi111 ceph-mon[24802]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 56 KiB/s, 0 objects/s recovering 2024-04-07T14:06:07.606 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-07T14:06:07.606 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:07 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:07.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-07T14:06:07.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:07 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:08.649 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:08 smithi145 ceph-mon[26800]: Deploying daemon osd.6 on smithi145 2024-04-07T14:06:08.650 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:08 smithi145 ceph-mon[26800]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:08 smithi111 ceph-mon[24802]: Deploying daemon osd.6 on smithi145 2024-04-07T14:06:08.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:08 smithi111 ceph-mon[24802]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:10.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:10 smithi111 ceph-mon[24802]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:10.986 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:10 smithi145 ceph-mon[26800]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:12.877 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:12 smithi145 ceph-mon[26800]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:12.878 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:12.878 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:12.878 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:12.878 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:12.962 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:12 smithi111 ceph-mon[24802]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:12.962 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:12.962 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:12.962 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:12.963 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:14.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:14 smithi145 ceph-mon[26800]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:14 smithi111 ceph-mon[24802]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:15.506 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 6 on host 'smithi145' 2024-04-07T14:06:15.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:15 smithi145 ceph-mon[26800]: from='osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T14:06:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:15 smithi145 ceph-mon[26800]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T14:06:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:15 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:15.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:15 smithi111 ceph-mon[24802]: from='osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T14:06:15.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:15 smithi111 ceph-mon[24802]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-07T14:06:15.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:15 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:16.176 DEBUG:teuthology.orchestra.run.smithi145:osd.6> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.6.service 2024-04-07T14:06:16.178 INFO:tasks.cephadm:Deploying osd.7 on smithi145 with /dev/vg_nvme/lv_1... 2024-04-07T14:06:16.179 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: osdmap e45: 7 total, 6 up, 7 in 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: from='osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:16.764 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-mon[26800]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:16.764 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:06:16.627+0000 7f29014eb700 -1 osd.6 0 waiting for initial osdmap 2024-04-07T14:06:16.764 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:06:16 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:06:16.634+0000 7f28fbe82700 -1 osd.6 46 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: osdmap e45: 7 total, 6 up, 7 in 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: from='osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:16.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:16 smithi111 ceph-mon[24802]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: osdmap e46: 7 total, 6 up, 7 in 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648] boot 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: osdmap e47: 7 total, 7 up, 7 in 2024-04-07T14:06:17.914 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:17 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: osdmap e46: 7 total, 6 up, 7 in 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: osd.6 [v2:172.21.15.145:6816/3557523648,v1:172.21.15.145:6817/3557523648] boot 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: osdmap e47: 7 total, 7 up, 7 in 2024-04-07T14:06:17.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:17 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:06:18.732 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-07T14:06:18.733 INFO:teuthology.orchestra.run.smithi145.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-07T14:06:18.733 INFO:teuthology.orchestra.run.smithi145.stderr: stderr: 10+0 records in 2024-04-07T14:06:18.733 INFO:teuthology.orchestra.run.smithi145.stderr:10+0 records out 2024-04-07T14:06:18.733 INFO:teuthology.orchestra.run.smithi145.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012863 s, 815 MB/s 2024-04-07T14:06:18.733 INFO:teuthology.orchestra.run.smithi145.stderr:--> Zapping successful for: 2024-04-07T14:06:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:18 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:06:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:18 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:06:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:18 smithi111 ceph-mon[24802]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:18 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:18.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:18 smithi111 ceph-mon[24802]: osdmap e48: 7 total, 7 up, 7 in 2024-04-07T14:06:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:18 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:06:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:18 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:06:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:18 smithi145 ceph-mon[26800]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 35 MiB used, 536 GiB / 536 GiB avail 2024-04-07T14:06:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:18 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:18 smithi145 ceph-mon[26800]: osdmap e48: 7 total, 7 up, 7 in 2024-04-07T14:06:19.949 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch daemon add osd smithi145:vg_nvme/lv_1 2024-04-07T14:06:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:20 smithi111 ceph-mon[24802]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:20 smithi111 ceph-mon[24802]: osdmap e49: 7 total, 7 up, 7 in 2024-04-07T14:06:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:20 smithi111 ceph-mon[24802]: osdmap e50: 7 total, 7 up, 7 in 2024-04-07T14:06:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:20 smithi145 ceph-mon[26800]: pgmap v147: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:20 smithi145 ceph-mon[26800]: osdmap e49: 7 total, 7 up, 7 in 2024-04-07T14:06:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:20 smithi145 ceph-mon[26800]: osdmap e50: 7 total, 7 up, 7 in 2024-04-07T14:06:22.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:22 smithi145 ceph-mon[26800]: osdmap e51: 7 total, 7 up, 7 in 2024-04-07T14:06:22.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:22 smithi145 ceph-mon[26800]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:22 smithi111 ceph-mon[24802]: osdmap e51: 7 total, 7 up, 7 in 2024-04-07T14:06:22.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:22 smithi111 ceph-mon[24802]: pgmap v151: 1 pgs: 1 remapped+peering; 577 KiB data, 40 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:23.477 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:23 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.523 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:23.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:23.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:23.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:23.524 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:23 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:24.486 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:24 smithi145 ceph-mon[26800]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:06:24.486 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:24 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi145 2024-04-07T14:06:24.486 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:24 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi145 to 5312M 2024-04-07T14:06:24.486 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:24 smithi145 ceph-mon[26800]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-07T14:06:24.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:24 smithi111 ceph-mon[24802]: from='client.24191 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi145:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:06:24.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:24 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi145 2024-04-07T14:06:24.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:24 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi145 to 5312M 2024-04-07T14:06:24.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:24 smithi111 ceph-mon[24802]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 105 KiB/s, 0 objects/s recovering 2024-04-07T14:06:25.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:25 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/3866365763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]: dispatch 2024-04-07T14:06:25.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:25 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]: dispatch 2024-04-07T14:06:25.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:25 smithi111 ceph-mon[24802]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]': finished 2024-04-07T14:06:25.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:25 smithi111 ceph-mon[24802]: osdmap e52: 8 total, 7 up, 8 in 2024-04-07T14:06:25.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:25 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:25 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/3866365763' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]: dispatch 2024-04-07T14:06:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:25 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]: dispatch 2024-04-07T14:06:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:25 smithi145 ceph-mon[26800]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "af49d895-c052-4107-9a78-5cd07c747514"}]': finished 2024-04-07T14:06:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:25 smithi145 ceph-mon[26800]: osdmap e52: 8 total, 7 up, 8 in 2024-04-07T14:06:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:25 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:26 smithi111 ceph-mon[24802]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-07T14:06:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:26 smithi111 ceph-mon[24802]: from='client.? 172.21.15.145:0/632658693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:06:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:26 smithi145 ceph-mon[26800]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 87 KiB/s, 0 objects/s recovering 2024-04-07T14:06:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:26 smithi145 ceph-mon[26800]: from='client.? 172.21.15.145:0/632658693' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-07T14:06:28.654 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:28 smithi145 ceph-mon[26800]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-07T14:06:28.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:28 smithi111 ceph-mon[24802]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-07T14:06:30.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:30 smithi145 ceph-mon[26800]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-07T14:06:30.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:30 smithi111 ceph-mon[24802]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-07T14:06:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:32 smithi145 ceph-mon[26800]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-07T14:06:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:32 smithi145 ceph-mon[26800]: Deploying daemon osd.7 on smithi145 2024-04-07T14:06:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:32 smithi111 ceph-mon[24802]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-07T14:06:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:32 smithi111 ceph-mon[24802]: Deploying daemon osd.7 on smithi145 2024-04-07T14:06:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:34 smithi145 ceph-mon[26800]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:34.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:34 smithi111 ceph-mon[24802]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:36.677 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:36 smithi145 ceph-mon[26800]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:36.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:36 smithi111 ceph-mon[24802]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:38.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:38.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:38.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:38.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:38.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:38.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:06:38.227 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:06:38.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:06:39.881 INFO:teuthology.orchestra.run.smithi145.stdout:Created osd(s) 7 on host 'smithi145' 2024-04-07T14:06:40.501 DEBUG:teuthology.orchestra.run.smithi145:osd.7> sudo journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.7.service 2024-04-07T14:06:40.503 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-04-07T14:06:40.504 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-07T14:06:40.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:40 smithi145 ceph-mon[26800]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:40.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:40 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:40.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:40 smithi145 ceph-mon[26800]: from='osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T14:06:40.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:40 smithi145 ceph-mon[26800]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T14:06:40.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:40 smithi111 ceph-mon[24802]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:40.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:40.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:40 smithi111 ceph-mon[24802]: from='osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T14:06:40.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:40 smithi111 ceph-mon[24802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-07T14:06:42.131 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: osdmap e53: 8 total, 7 up, 8 in 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: from='osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:42.132 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:41 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:42.180 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-07T14:06:42.180 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: osdmap e53: 8 total, 7 up, 8 in 2024-04-07T14:06:42.180 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:42.181 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: from='osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:42.181 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]: dispatch 2024-04-07T14:06:42.181 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:42.181 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:42.181 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:06:41.888+0000 7f27033b8700 -1 osd.7 0 waiting for initial osdmap 2024-04-07T14:06:42.181 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:06:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:06:41.895+0000 7f26ff552700 -1 osd.7 54 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-07T14:06:42.530 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:06:43.115 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":54,"num_osds":8,"num_up_osds":7,"osd_up_since":1712498777,"num_in_osds":8,"osd_in_since":1712498785,"num_remapped_pgs":0} 2024-04-07T14:06:43.173 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:42 smithi145 ceph-mon[26800]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:06:43.173 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:42 smithi145 ceph-mon[26800]: osdmap e54: 8 total, 7 up, 8 in 2024-04-07T14:06:43.173 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:43.173 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:42 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:43.173 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:42 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2497412861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T14:06:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:42 smithi111 ceph-mon[24802]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi145", "root=default"]}]': finished 2024-04-07T14:06:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:42 smithi111 ceph-mon[24802]: osdmap e54: 8 total, 7 up, 8 in 2024-04-07T14:06:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:42 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:43.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:42 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2497412861' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T14:06:44.117 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd stat -f json 2024-04-07T14:06:44.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: purged_snaps scrub starts 2024-04-07T14:06:44.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: purged_snaps scrub ok 2024-04-07T14:06:44.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177] boot 2024-04-07T14:06:44.145 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: osdmap e55: 8 total, 8 up, 8 in 2024-04-07T14:06:44.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:44.146 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:43 smithi111 ceph-mon[24802]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:44.247 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: purged_snaps scrub starts 2024-04-07T14:06:44.247 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: purged_snaps scrub ok 2024-04-07T14:06:44.247 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: osd.7 [v2:172.21.15.145:6824/1937789177,v1:172.21.15.145:6825/1937789177] boot 2024-04-07T14:06:44.248 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: osdmap e55: 8 total, 8 up, 8 in 2024-04-07T14:06:44.248 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:06:44.248 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:43 smithi145 ceph-mon[26800]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 41 MiB used, 626 GiB / 626 GiB avail 2024-04-07T14:06:44.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:44 smithi111 ceph-mon[24802]: osdmap e56: 8 total, 8 up, 8 in 2024-04-07T14:06:45.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:44 smithi145 ceph-mon[26800]: osdmap e56: 8 total, 8 up, 8 in 2024-04-07T14:06:46.143 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:06:46.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: osdmap e57: 8 total, 8 up, 8 in 2024-04-07T14:06:46.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: Detected new or changed devices on smithi145 2024-04-07T14:06:46.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.143 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: Adjusting osd_memory_target on smithi145 to 3984M 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: osdmap e58: 8 total, 8 up, 8 in 2024-04-07T14:06:46.144 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:45 smithi111 ceph-mon[24802]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: osdmap e57: 8 total, 8 up, 8 in 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: Detected new or changed devices on smithi145 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: Adjusting osd_memory_target on smithi145 to 3984M 2024-04-07T14:06:46.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: osdmap e58: 8 total, 8 up, 8 in 2024-04-07T14:06:46.274 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:45 smithi145 ceph-mon[26800]: pgmap v170: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:06:46.738 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":58,"num_osds":8,"num_up_osds":8,"osd_up_since":1712498802,"num_in_osds":8,"osd_in_since":1712498785,"num_remapped_pgs":0} 2024-04-07T14:06:46.738 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T14:06:46.993 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:06:47.115 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:46 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3714746065' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T14:06:47.115 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:46 smithi111 ceph-mon[24802]: osdmap e59: 8 total, 8 up, 8 in 2024-04-07T14:06:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:46 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3714746065' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-07T14:06:47.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:46 smithi145 ceph-mon[26800]: osdmap e59: 8 total, 8 up, 8 in 2024-04-07T14:06:48.137 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:47 smithi111 ceph-mon[24802]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:06:48.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:47 smithi145 ceph-mon[26800]: pgmap v172: 1 pgs: 1 remapped+peering; 577 KiB data, 47 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:06:48.622 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:06:48.622 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":59,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","created":"2024-04-07T14:00:50.660173+0000","modified":"2024-04-07T14:06:46.361077+0000","last_up_change":"2024-04-07T14:06:42.886563+0000","last_in_change":"2024-04-07T14:06:25.224655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T14:04:41.834336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f4937d42-740b-4c86-8497-4f8d55b1c30b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6803","nonce":2388610062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6805","nonce":2388610062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6809","nonce":2388610062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6807","nonce":2388610062}]},"public_addr":"172.21.15.111:6803/2388610062","cluster_addr":"172.21.15.111:6805/2388610062","heartbeat_back_addr":"172.21.15.111:6809/2388610062","heartbeat_front_addr":"172.21.15.111:6807/2388610062","state":["exists","up"]},{"osd":1,"uuid":"d0ebc0e4-c2a4-49ef-bf0c-43bc03873463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6811","nonce":1959509298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6813","nonce":1959509298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6817","nonce":1959509298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6815","nonce":1959509298}]},"public_addr":"172.21.15.111:6811/1959509298","cluster_addr":"172.21.15.111:6813/1959509298","heartbeat_back_addr":"172.21.15.111:6817/1959509298","heartbeat_front_addr":"172.21.15.111:6815/1959509298","state":["exists","up"]},{"osd":2,"uuid":"ed8d6081-0935-4cab-987b-7393b5e15e12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6819","nonce":2995507539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6821","nonce":2995507539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6825","nonce":2995507539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6823","nonce":2995507539}]},"public_addr":"172.21.15.111:6819/2995507539","cluster_addr":"172.21.15.111:6821/2995507539","heartbeat_back_addr":"172.21.15.111:6825/2995507539","heartbeat_front_addr":"172.21.15.111:6823/2995507539","state":["exists","up"]},{"osd":3,"uuid":"febc59b7-eb37-4e44-9339-46400b69a401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6827","nonce":1526297577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6829","nonce":1526297577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6833","nonce":1526297577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6831","nonce":1526297577}]},"public_addr":"172.21.15.111:6827/1526297577","cluster_addr":"172.21.15.111:6829/1526297577","heartbeat_back_addr":"172.21.15.111:6833/1526297577","heartbeat_front_addr":"172.21.15.111:6831/1526297577","state":["exists","up"]},{"osd":4,"uuid":"6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6801","nonce":3801945651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6803","nonce":3801945651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6807","nonce":3801945651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6805","nonce":3801945651}]},"public_addr":"172.21.15.145:6801/3801945651","cluster_addr":"172.21.15.145:6803/3801945651","heartbeat_back_addr":"172.21.15.145:6807/3801945651","heartbeat_front_addr":"172.21.15.145:6805/3801945651","state":["exists","up"]},{"osd":5,"uuid":"4fc8f366-9ee3-4b85-961d-db120af9f964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6809","nonce":2218575709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6811","nonce":2218575709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6815","nonce":2218575709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6813","nonce":2218575709}]},"public_addr":"172.21.15.145:6809/2218575709","cluster_addr":"172.21.15.145:6811/2218575709","heartbeat_back_addr":"172.21.15.145:6815/2218575709","heartbeat_front_addr":"172.21.15.145:6813/2218575709","state":["exists","up"]},{"osd":6,"uuid":"5a86e7a8-01c0-4188-9774-cc71681e8664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6817","nonce":3557523648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6819","nonce":3557523648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6823","nonce":3557523648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6821","nonce":3557523648}]},"public_addr":"172.21.15.145:6817/3557523648","cluster_addr":"172.21.15.145:6819/3557523648","heartbeat_back_addr":"172.21.15.145:6823/3557523648","heartbeat_front_addr":"172.21.15.145:6821/3557523648","state":["exists","up"]},{"osd":7,"uuid":"af49d895-c052-4107-9a78-5cd07c747514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6825","nonce":1937789177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6827","nonce":1937789177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6831","nonce":1937789177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6829","nonce":1937789177}]},"public_addr":"172.21.15.145:6825/1937789177","cluster_addr":"172.21.15.145:6827/1937789177","heartbeat_back_addr":"172.21.15.145:6831/1937789177","heartbeat_front_addr":"172.21.15.145:6829/1937789177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:03:41.094634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:12.535296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:39.019598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:04.640986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:26.413204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:50.870753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:16.255152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:41.114886+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/788378731":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/370719533":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/3087529383":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/43409346":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/1266830854":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/271928614":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6800/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:6800/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/2607680680":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6801/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6800/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1426862294":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6801/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1962951409":"2024-04-08T14:01:18.107772+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T14:06:49.211 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:48 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/4004803300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:06:49.234 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-07T14:04:41.834336+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '22', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-04-07T14:06:49.234 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd pool get .mgr pg_num 2024-04-07T14:06:49.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:48 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/4004803300' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:06:49.446 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:06:50.226 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:49 smithi111 ceph-mon[24802]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-07T14:06:50.272 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:49 smithi145 ceph-mon[26800]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 91 KiB/s, 0 objects/s recovering 2024-04-07T14:06:51.255 INFO:teuthology.orchestra.run.smithi111.stdout:pg_num: 1 2024-04-07T14:06:51.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:51 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/879798118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-07T14:06:51.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:51 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/879798118' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-07T14:06:52.144 INFO:tasks.cephadm:Setting up client nodes... 2024-04-07T14:06:52.145 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-07T14:06:52.451 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:52 smithi111 ceph-mon[24802]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-07T14:06:52.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:52 smithi145 ceph-mon[26800]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-04-07T14:06:54.105 INFO:teuthology.orchestra.run.smithi111.stdout:[client.0] 2024-04-07T14:06:54.106 INFO:teuthology.orchestra.run.smithi111.stdout: key = AQB+qBJmH0smBhAAMXXAbqhyEvdEFLBZmd6lSA== 2024-04-07T14:06:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:54 smithi111 ceph-mon[24802]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-07T14:06:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:54 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1326701920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T14:06:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:54 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1326701920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T14:06:54.782 DEBUG:teuthology.orchestra.run.smithi111:> set -ex 2024-04-07T14:06:54.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-07T14:06:54.782 DEBUG:teuthology.orchestra.run.smithi111:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-07T14:06:54.825 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-07T14:06:54.825 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-07T14:06:54.826 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph mgr dump --format=json 2024-04-07T14:06:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:54 smithi145 ceph-mon[26800]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-07T14:06:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1326701920' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-07T14:06:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:54 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1326701920' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-07T14:06:55.077 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:06:56.506 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:56 smithi111 ceph-mon[24802]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:56 smithi145 ceph-mon[26800]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-04-07T14:06:56.948 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:06:57.523 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:57 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/812298353' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-07T14:06:57.731 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":19,"active_gid":14200,"active_name":"smithi111.rtrlso","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6800","nonce":3579713956},{"type":"v1","addr":"172.21.15.111:6801","nonce":3579713956}]},"active_addr":"172.21.15.111:6801/3579713956","active_change":"2024-04-07T14:02:37.462283+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":14218,"name":"smithi145.qiunpb","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.23.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:8.3.5","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"docker.io/library/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"docker.io/arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.3.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.33.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.111:8443/","prometheus":"http://172.21.15.111:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":5,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1686774199}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3350837090}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":3072256539}]},{"addrvec":[{"type":"v2","addr":"172.21.15.111:0","nonce":1206980144}]}]}} 2024-04-07T14:06:57.737 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-07T14:06:57.737 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-07T14:06:57.737 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T14:06:57.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:57 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/812298353' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-07T14:06:57.952 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:06:58.629 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:06:58 smithi111 ceph-mon[24802]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-04-07T14:06:58.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:06:58 smithi145 ceph-mon[26800]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 46 KiB/s, 0 objects/s recovering 2024-04-07T14:06:59.617 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:06:59.617 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":59,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","created":"2024-04-07T14:00:50.660173+0000","modified":"2024-04-07T14:06:46.361077+0000","last_up_change":"2024-04-07T14:06:42.886563+0000","last_in_change":"2024-04-07T14:06:25.224655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T14:04:41.834336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f4937d42-740b-4c86-8497-4f8d55b1c30b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6803","nonce":2388610062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6805","nonce":2388610062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6809","nonce":2388610062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6807","nonce":2388610062}]},"public_addr":"172.21.15.111:6803/2388610062","cluster_addr":"172.21.15.111:6805/2388610062","heartbeat_back_addr":"172.21.15.111:6809/2388610062","heartbeat_front_addr":"172.21.15.111:6807/2388610062","state":["exists","up"]},{"osd":1,"uuid":"d0ebc0e4-c2a4-49ef-bf0c-43bc03873463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6811","nonce":1959509298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6813","nonce":1959509298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6817","nonce":1959509298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6815","nonce":1959509298}]},"public_addr":"172.21.15.111:6811/1959509298","cluster_addr":"172.21.15.111:6813/1959509298","heartbeat_back_addr":"172.21.15.111:6817/1959509298","heartbeat_front_addr":"172.21.15.111:6815/1959509298","state":["exists","up"]},{"osd":2,"uuid":"ed8d6081-0935-4cab-987b-7393b5e15e12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6819","nonce":2995507539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6821","nonce":2995507539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6825","nonce":2995507539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6823","nonce":2995507539}]},"public_addr":"172.21.15.111:6819/2995507539","cluster_addr":"172.21.15.111:6821/2995507539","heartbeat_back_addr":"172.21.15.111:6825/2995507539","heartbeat_front_addr":"172.21.15.111:6823/2995507539","state":["exists","up"]},{"osd":3,"uuid":"febc59b7-eb37-4e44-9339-46400b69a401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6827","nonce":1526297577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6829","nonce":1526297577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6833","nonce":1526297577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6831","nonce":1526297577}]},"public_addr":"172.21.15.111:6827/1526297577","cluster_addr":"172.21.15.111:6829/1526297577","heartbeat_back_addr":"172.21.15.111:6833/1526297577","heartbeat_front_addr":"172.21.15.111:6831/1526297577","state":["exists","up"]},{"osd":4,"uuid":"6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6801","nonce":3801945651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6803","nonce":3801945651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6807","nonce":3801945651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6805","nonce":3801945651}]},"public_addr":"172.21.15.145:6801/3801945651","cluster_addr":"172.21.15.145:6803/3801945651","heartbeat_back_addr":"172.21.15.145:6807/3801945651","heartbeat_front_addr":"172.21.15.145:6805/3801945651","state":["exists","up"]},{"osd":5,"uuid":"4fc8f366-9ee3-4b85-961d-db120af9f964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6809","nonce":2218575709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6811","nonce":2218575709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6815","nonce":2218575709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6813","nonce":2218575709}]},"public_addr":"172.21.15.145:6809/2218575709","cluster_addr":"172.21.15.145:6811/2218575709","heartbeat_back_addr":"172.21.15.145:6815/2218575709","heartbeat_front_addr":"172.21.15.145:6813/2218575709","state":["exists","up"]},{"osd":6,"uuid":"5a86e7a8-01c0-4188-9774-cc71681e8664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6817","nonce":3557523648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6819","nonce":3557523648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6823","nonce":3557523648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6821","nonce":3557523648}]},"public_addr":"172.21.15.145:6817/3557523648","cluster_addr":"172.21.15.145:6819/3557523648","heartbeat_back_addr":"172.21.15.145:6823/3557523648","heartbeat_front_addr":"172.21.15.145:6821/3557523648","state":["exists","up"]},{"osd":7,"uuid":"af49d895-c052-4107-9a78-5cd07c747514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6825","nonce":1937789177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6827","nonce":1937789177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6831","nonce":1937789177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6829","nonce":1937789177}]},"public_addr":"172.21.15.145:6825/1937789177","cluster_addr":"172.21.15.145:6827/1937789177","heartbeat_back_addr":"172.21.15.145:6831/1937789177","heartbeat_front_addr":"172.21.15.145:6829/1937789177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:03:41.094634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:12.535296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:39.019598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:04.640986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:26.413204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:50.870753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:16.255152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:41.114886+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/788378731":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/370719533":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/3087529383":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/43409346":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/1266830854":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/271928614":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6800/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:6800/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/2607680680":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6801/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6800/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1426862294":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6801/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1962951409":"2024-04-08T14:01:18.107772+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T14:07:00.328 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-07T14:07:00.328 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd dump --format=json 2024-04-07T14:07:00.542 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:00.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:00 smithi111 ceph-mon[24802]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-07T14:07:00.865 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:00 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2442611445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:07:01.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:00 smithi145 ceph-mon[26800]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail; 43 KiB/s, 0 objects/s recovering 2024-04-07T14:07:01.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:00 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2442611445' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:07:02.132 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:07:02.132 INFO:teuthology.orchestra.run.smithi111.stdout:{"epoch":59,"fsid":"2bfa0756-f4e7-11ee-b649-cb9ed24678a4","created":"2024-04-07T14:00:50.660173+0000","modified":"2024-04-07T14:06:46.361077+0000","last_up_change":"2024-04-07T14:06:42.886563+0000","last_in_change":"2024-04-07T14:06:25.224655+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-07T14:04:41.834336+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"22","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"f4937d42-740b-4c86-8497-4f8d55b1c30b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":57,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6802","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6803","nonce":2388610062}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6804","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6805","nonce":2388610062}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6808","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6809","nonce":2388610062}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6806","nonce":2388610062},{"type":"v1","addr":"172.21.15.111:6807","nonce":2388610062}]},"public_addr":"172.21.15.111:6803/2388610062","cluster_addr":"172.21.15.111:6805/2388610062","heartbeat_back_addr":"172.21.15.111:6809/2388610062","heartbeat_front_addr":"172.21.15.111:6807/2388610062","state":["exists","up"]},{"osd":1,"uuid":"d0ebc0e4-c2a4-49ef-bf0c-43bc03873463","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6810","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6811","nonce":1959509298}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6812","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6813","nonce":1959509298}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6816","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6817","nonce":1959509298}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6814","nonce":1959509298},{"type":"v1","addr":"172.21.15.111:6815","nonce":1959509298}]},"public_addr":"172.21.15.111:6811/1959509298","cluster_addr":"172.21.15.111:6813/1959509298","heartbeat_back_addr":"172.21.15.111:6817/1959509298","heartbeat_front_addr":"172.21.15.111:6815/1959509298","state":["exists","up"]},{"osd":2,"uuid":"ed8d6081-0935-4cab-987b-7393b5e15e12","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6818","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6819","nonce":2995507539}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6820","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6821","nonce":2995507539}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6824","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6825","nonce":2995507539}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6822","nonce":2995507539},{"type":"v1","addr":"172.21.15.111:6823","nonce":2995507539}]},"public_addr":"172.21.15.111:6819/2995507539","cluster_addr":"172.21.15.111:6821/2995507539","heartbeat_back_addr":"172.21.15.111:6825/2995507539","heartbeat_front_addr":"172.21.15.111:6823/2995507539","state":["exists","up"]},{"osd":3,"uuid":"febc59b7-eb37-4e44-9339-46400b69a401","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6826","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6827","nonce":1526297577}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6828","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6829","nonce":1526297577}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6832","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6833","nonce":1526297577}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.111:6830","nonce":1526297577},{"type":"v1","addr":"172.21.15.111:6831","nonce":1526297577}]},"public_addr":"172.21.15.111:6827/1526297577","cluster_addr":"172.21.15.111:6829/1526297577","heartbeat_back_addr":"172.21.15.111:6833/1526297577","heartbeat_front_addr":"172.21.15.111:6831/1526297577","state":["exists","up"]},{"osd":4,"uuid":"6e3a5ae3-b81b-45f7-9a1d-c8594b01ffce","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6800","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6801","nonce":3801945651}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6802","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6803","nonce":3801945651}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6806","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6807","nonce":3801945651}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6804","nonce":3801945651},{"type":"v1","addr":"172.21.15.145:6805","nonce":3801945651}]},"public_addr":"172.21.15.145:6801/3801945651","cluster_addr":"172.21.15.145:6803/3801945651","heartbeat_back_addr":"172.21.15.145:6807/3801945651","heartbeat_front_addr":"172.21.15.145:6805/3801945651","state":["exists","up"]},{"osd":5,"uuid":"4fc8f366-9ee3-4b85-961d-db120af9f964","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6808","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6809","nonce":2218575709}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6810","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6811","nonce":2218575709}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6814","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6815","nonce":2218575709}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6812","nonce":2218575709},{"type":"v1","addr":"172.21.15.145:6813","nonce":2218575709}]},"public_addr":"172.21.15.145:6809/2218575709","cluster_addr":"172.21.15.145:6811/2218575709","heartbeat_back_addr":"172.21.15.145:6815/2218575709","heartbeat_front_addr":"172.21.15.145:6813/2218575709","state":["exists","up"]},{"osd":6,"uuid":"5a86e7a8-01c0-4188-9774-cc71681e8664","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6816","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6817","nonce":3557523648}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6818","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6819","nonce":3557523648}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6822","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6823","nonce":3557523648}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6820","nonce":3557523648},{"type":"v1","addr":"172.21.15.145:6821","nonce":3557523648}]},"public_addr":"172.21.15.145:6817/3557523648","cluster_addr":"172.21.15.145:6819/3557523648","heartbeat_back_addr":"172.21.15.145:6823/3557523648","heartbeat_front_addr":"172.21.15.145:6821/3557523648","state":["exists","up"]},{"osd":7,"uuid":"af49d895-c052-4107-9a78-5cd07c747514","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":55,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6824","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6825","nonce":1937789177}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6826","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6827","nonce":1937789177}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6830","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6831","nonce":1937789177}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.145:6828","nonce":1937789177},{"type":"v1","addr":"172.21.15.145:6829","nonce":1937789177}]},"public_addr":"172.21.15.145:6825/1937789177","cluster_addr":"172.21.15.145:6827/1937789177","heartbeat_back_addr":"172.21.15.145:6831/1937789177","heartbeat_front_addr":"172.21.15.145:6829/1937789177","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:03:41.094634+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:12.535296+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:04:39.019598+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:04.640986+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:26.413204+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:05:50.870753+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:16.255152+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2024-04-07T14:06:41.114886+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.111:6801/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/788378731":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/370719533":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/3087529383":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/43409346":"2024-04-08T14:02:37.462002+0000","172.21.15.111:0/1266830854":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/271928614":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6800/526319469":"2024-04-08T14:02:37.462002+0000","172.21.15.111:6800/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:0/2607680680":"2024-04-08T14:01:18.107772+0000","172.21.15.111:6801/1155077031":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6800/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1426862294":"2024-04-08T14:01:47.640808+0000","172.21.15.111:6801/867316678":"2024-04-08T14:01:18.107772+0000","172.21.15.111:0/1962951409":"2024-04-08T14:01:18.107772+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-07T14:07:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:02 smithi145 ceph-mon[26800]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:02 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3880041365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:07:02.869 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:02 smithi111 ceph-mon[24802]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:02.869 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:02 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3880041365' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-07T14:07:02.892 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.0 flush_pg_stats 2024-04-07T14:07:02.893 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.1 flush_pg_stats 2024-04-07T14:07:02.893 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.2 flush_pg_stats 2024-04-07T14:07:02.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.3 flush_pg_stats 2024-04-07T14:07:02.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.4 flush_pg_stats 2024-04-07T14:07:02.894 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.5 flush_pg_stats 2024-04-07T14:07:02.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.6 flush_pg_stats 2024-04-07T14:07:02.895 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph tell osd.7 flush_pg_stats 2024-04-07T14:07:03.283 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:03.293 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:03.861 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:03.862 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:04.291 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:04.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:04 smithi111 ceph-mon[24802]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:05.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:04 smithi145 ceph-mon[26800]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:05.325 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:05.327 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:06.054 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:06.456 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:06 smithi111 ceph-mon[24802]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:06.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:06 smithi145 ceph-mon[26800]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:07.695 INFO:teuthology.orchestra.run.smithi111.stdout:201863462924 2024-04-07T14:07:07.695 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.6 2024-04-07T14:07:08.823 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:08 smithi111 ceph-mon[24802]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:08 smithi145 ceph-mon[26800]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:10.066 INFO:teuthology.orchestra.run.smithi111.stdout:236223201287 2024-04-07T14:07:10.067 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.7 2024-04-07T14:07:10.636 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:10 smithi111 ceph-mon[24802]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:11.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:10 smithi145 ceph-mon[26800]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:12.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:12 smithi111 ceph-mon[24802]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:12 smithi145 ceph-mon[26800]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:14.106 INFO:teuthology.orchestra.run.smithi111.stdout:81604378657 2024-04-07T14:07:14.107 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.2 2024-04-07T14:07:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:14 smithi111 ceph-mon[24802]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:14 smithi145 ceph-mon[26800]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:15.793 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:15.908 INFO:teuthology.orchestra.run.smithi111.stdout:60129542183 2024-04-07T14:07:15.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.1 2024-04-07T14:07:16.372 INFO:teuthology.orchestra.run.smithi111.stdout:133143986200 2024-04-07T14:07:16.372 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.4 2024-04-07T14:07:16.395 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:16 smithi111 ceph-mon[24802]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:16.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:16 smithi145 ceph-mon[26800]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:17.344 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:17.345 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:17.867 INFO:teuthology.orchestra.run.smithi111.stdout:167503724562 2024-04-07T14:07:17.867 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.5 2024-04-07T14:07:18.266 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:18.716 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:18 smithi111 ceph-mon[24802]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:18 smithi145 ceph-mon[26800]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:19.672 INFO:teuthology.orchestra.run.smithi111.stdout:111669149726 2024-04-07T14:07:19.672 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.3 2024-04-07T14:07:20.042 INFO:teuthology.orchestra.run.smithi111.stdout:38654705709 2024-04-07T14:07:20.042 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph osd last-stat-seq osd.0 2024-04-07T14:07:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:20 smithi111 ceph-mon[24802]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:20 smithi145 ceph-mon[26800]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:22.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:22 smithi111 ceph-mon[24802]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:22 smithi145 ceph-mon[26800]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:23.765 INFO:teuthology.orchestra.run.smithi111.stdout:81604378659 2024-04-07T14:07:24.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:24 smithi111 ceph-mon[24802]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:24.889 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:24 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/208817113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-07T14:07:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:24 smithi145 ceph-mon[26800]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:24 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/208817113' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-07T14:07:25.347 INFO:teuthology.orchestra.run.smithi111.stdout:236223201290 2024-04-07T14:07:25.538 INFO:teuthology.orchestra.run.smithi111.stdout:167503724564 2024-04-07T14:07:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:25 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1355233594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-07T14:07:25.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:25 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2381177520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-07T14:07:25.796 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:25 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1355233594' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-07T14:07:25.796 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:25 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2381177520' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-07T14:07:26.195 INFO:tasks.cephadm.ceph_manager.ceph:need seq 236223201287 got 236223201290 for osd.7 2024-04-07T14:07:26.196 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:26.660 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724562 got 167503724564 for osd.5 2024-04-07T14:07:26.661 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:26.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:26 smithi111 ceph-mon[24802]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:27.020 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378657 got 81604378659 for osd.2 2024-04-07T14:07:27.020 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:27.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:26 smithi145 ceph-mon[26800]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:27.034 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:27.036 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:27.037 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:27.039 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:27.750 INFO:teuthology.orchestra.run.smithi111.stdout:201863462928 2024-04-07T14:07:28.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:28 smithi111 ceph-mon[24802]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:28.975 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:28 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/456685949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-07T14:07:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:28 smithi145 ceph-mon[26800]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:28 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/456685949' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-07T14:07:30.909 INFO:teuthology.orchestra.run.smithi111.stdout:60129542186 2024-04-07T14:07:30.909 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:30 smithi111 ceph-mon[24802]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:31.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:30 smithi145 ceph-mon[26800]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:31.490 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462924 got 201863462928 for osd.6 2024-04-07T14:07:31.490 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:31.924 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:31 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2349328421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-07T14:07:31.945 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:31 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2349328421' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-07T14:07:32.228 INFO:teuthology.orchestra.run.smithi111.stdout:111669149728 2024-04-07T14:07:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:32 smithi111 ceph-mon[24802]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:32.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:32 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3710694851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-07T14:07:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:32 smithi145 ceph-mon[26800]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:32 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3710694851' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-07T14:07:33.378 INFO:teuthology.orchestra.run.smithi111.stdout:38654705712 2024-04-07T14:07:33.712 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149726 got 111669149728 for osd.3 2024-04-07T14:07:33.712 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:33 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/4201102720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-07T14:07:34.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:33 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/4201102720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-07T14:07:34.089 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542183 got 60129542186 for osd.1 2024-04-07T14:07:34.089 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:34.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:34 smithi111 ceph-mon[24802]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:34.933 INFO:teuthology.orchestra.run.smithi111.stdout:133143986204 2024-04-07T14:07:35.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:34 smithi145 ceph-mon[26800]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:35.121 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705709 got 38654705712 for osd.0 2024-04-07T14:07:35.121 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:35.735 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:35 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1057400135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-07T14:07:35.758 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986200 got 133143986204 for osd.4 2024-04-07T14:07:35.758 DEBUG:teuthology.parallel:result is None 2024-04-07T14:07:35.759 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-07T14:07:35.759 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-07T14:07:35.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:35 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1057400135' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-07T14:07:35.973 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:36.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:36 smithi111 ceph-mon[24802]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:36 smithi145 ceph-mon[26800]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:37.707 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:07:37.707 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:07:37.707 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:07:37.709 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-07T14:07:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:07:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:07:38.258 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":197,"stamp":"2024-04-07T14:07:37.568038+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48872,"kb_used_data":6312,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912344,"statfs":{"total":767960285184,"available":767910240256,"internally_reserved":0,"allocated":6463488,"data_stored":3547530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002162"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-07T14:06:46.371380+0000","last_change":"2024-04-07T14:06:46.371380+0000","last_active":"2024-04-07T14:06:46.371380+0000","last_peered":"2024-04-07T14:06:46.371380+0000","last_clean":"2024-04-07T14:06:46.371380+0000","last_became_active":"2024-04-07T14:06:46.370997+0000","last_became_peered":"2024-04-07T14:06:46.370997+0000","last_unstale":"2024-04-07T14:06:46.371380+0000","last_undegraded":"2024-04-07T14:06:46.371380+0000","last_fullsized":"2024-04-07T14:06:46.371380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-07T14:04:41.834870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-07T14:04:41.834870+0000","last_clean_scrub_stamp":"2024-04-07T14:04:41.834870+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T22:06:47.350692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201293,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.626}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68200000000000005}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66400000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56299999999999994}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63300000000000001}]}]},{"osd":6,"up_from":47,"seq":201863462930,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.56000000000000005}]},{"osd":1,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.50700000000000001}]},{"osd":2,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.624},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.60399999999999998}]},{"osd":3,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.5},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.58699999999999997}]},{"osd":4,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.45300000000000001}]},{"osd":5,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.39700000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"min":{"1min":0.25,"5min":0.216,"15min":0.216},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.34100000000000003},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247}]},{"osd":2,"last update":"Sun Apr 7 14:06:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"min":{"1min":0.31900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.314,"5min":0.218,"15min":0.218},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.27700000000000002}]},{"osd":3,"last update":"Sun Apr 7 14:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Sun Apr 7 14:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76500000000000001,"5min":0.874,"15min":0.874},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.624}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705713,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 7 14:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.34100000000000003},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.625,"15min":0.625},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36099999999999999,"15min":0.34799999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47599999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.314}]},{"osd":2,"last update":"Sun Apr 7 14:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.254},{"interface":"front","average":{"1min":0.434,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.378}]},{"osd":3,"last update":"Sun Apr 7 14:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.437},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56000000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.52300000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.33900000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:06:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.59799999999999998,"15min":0.59799999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.93600000000000005,"5min":0.93600000000000005,"15min":0.93600000000000005},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.91000000000000003,"5min":0.91000000000000003,"15min":0.91000000000000003},"last":0.54100000000000004}]},{"osd":5,"last update":"Sun Apr 7 14:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.58699999999999997}]},{"osd":6,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.622}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68400000000000005}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.246},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.36399999999999999}]},{"osd":1,"last update":"Sun Apr 7 14:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.29999999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.27100000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.25600000000000001}]},{"osd":3,"last update":"Sun Apr 7 14:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.315},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.33800000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":1.04,"15min":1.04},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82399999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Sun Apr 7 14:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Sun Apr 7 14:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.315},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.27700000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71899999999999997,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.59199999999999997},"min":{"1min":0.45600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Apr 7 14:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.753,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.45400000000000001}]},{"osd":1,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71099999999999997,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.33800000000000002}]},{"osd":2,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71299999999999997,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.379},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39600000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.499}]},{"osd":3,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.41499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.48299999999999998}]},{"osd":5,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.56299999999999994}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.52100000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58799999999999997}]}]},{"osd":5,"up_from":39,"seq":167503724566,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.78700000000000003}]},{"osd":1,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.67500000000000004}]},{"osd":2,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.72299999999999998}]},{"osd":3,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.755}]},{"osd":4,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.53300000000000003}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.59599999999999997}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68899999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-07T14:07:38.259 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph pg dump --format=json 2024-04-07T14:07:38.474 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:38.886 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:38 smithi111 ceph-mon[24802]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:38.887 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:38 smithi111 ceph-mon[24802]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:07:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:38 smithi145 ceph-mon[26800]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:38 smithi145 ceph-mon[26800]: from='client.14442 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:07:40.172 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:07:40.173 INFO:teuthology.orchestra.run.smithi111.stderr:dumped all 2024-04-07T14:07:40.730 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:40 smithi111 ceph-mon[24802]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:40.752 INFO:teuthology.orchestra.run.smithi111.stdout:{"pg_ready":true,"pg_map":{"version":198,"stamp":"2024-04-07T14:07:39.568281+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":48872,"kb_used_data":6312,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":749912344,"statfs":{"total":767960285184,"available":767910240256,"internally_reserved":0,"allocated":6463488,"data_stored":3547530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002054"},"pg_stats":[{"pgid":"1.0","version":"22'117","reported_seq":226,"reported_epoch":59,"state":"active+clean","last_fresh":"2024-04-07T14:06:46.371380+0000","last_change":"2024-04-07T14:06:46.371380+0000","last_active":"2024-04-07T14:06:46.371380+0000","last_peered":"2024-04-07T14:06:46.371380+0000","last_clean":"2024-04-07T14:06:46.371380+0000","last_became_active":"2024-04-07T14:06:46.370997+0000","last_became_peered":"2024-04-07T14:06:46.370997+0000","last_unstale":"2024-04-07T14:06:46.371380+0000","last_undegraded":"2024-04-07T14:06:46.371380+0000","last_fullsized":"2024-04-07T14:06:46.371380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-07T14:04:41.834870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-07T14:04:41.834870+0000","last_clean_scrub_stamp":"2024-04-07T14:04:41.834870+0000","objects_scrubbed":0,"log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-08T22:06:47.350692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":55,"seq":236223201294,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6052,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":4928,"kb_avail":93739100,"statfs":{"total":95995035648,"available":95988838400,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5046272},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76600000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.752}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73999999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55000000000000004}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60399999999999998}]}]},{"osd":6,"up_from":47,"seq":201863462931,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6108,"kb_used_data":1108,"kb_used_omap":0,"kb_used_meta":4992,"kb_avail":93739044,"statfs":{"total":95995035648,"available":95988781056,"internally_reserved":0,"allocated":1134592,"data_stored":773573,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5111808},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.68200000000000005,"5min":0.68200000000000005,"15min":0.68200000000000005},"last":0.54500000000000004}]},{"osd":1,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.56799999999999995,"15min":0.56799999999999995},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.73399999999999999,"5min":0.73399999999999999,"15min":0.73399999999999999},"last":0.376},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.585,"5min":1.585,"15min":1.585},"last":0.52300000000000002}]},{"osd":2,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.57099999999999995,"15min":0.57099999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.48699999999999999}]},{"osd":3,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.438,"5min":0.438,"15min":0.438},"max":{"1min":0.69399999999999995,"5min":0.69399999999999995,"15min":0.69399999999999995},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"min":{"1min":0.46400000000000002,"5min":0.46400000000000002,"15min":0.46400000000000002},"max":{"1min":1.6259999999999999,"5min":1.6259999999999999,"15min":1.6259999999999999},"last":0.45900000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.48999999999999999,"15min":0.48999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.439}]},{"osd":5,"last update":"Sun Apr 7 14:07:21 2024","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.73599999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.499}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542188,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6364,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738788,"statfs":{"total":95995035648,"available":95988518912,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.35899999999999999,"15min":0.33300000000000002},"min":{"1min":0.25,"5min":0.216,"15min":0.216},"max":{"1min":1.3899999999999999,"5min":1.3899999999999999,"15min":1.3899999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.35799999999999998,"15min":0.34100000000000003},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.247}]},{"osd":2,"last update":"Sun Apr 7 14:06:44 2024","interfaces":[{"interface":"back","average":{"1min":0.42299999999999999,"5min":0.36099999999999999,"15min":0.35099999999999998},"min":{"1min":0.31900000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.39500000000000002,"15min":0.38200000000000001},"min":{"1min":0.314,"5min":0.218,"15min":0.218},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.27700000000000002}]},{"osd":3,"last update":"Sun Apr 7 14:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.44,"15min":0.436},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":1.264,"5min":1.264,"15min":1.264},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48599999999999999,"15min":0.48199999999999998},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.32000000000000001}]},{"osd":4,"last update":"Sun Apr 7 14:07:30 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.57399999999999995,"15min":0.57599999999999996},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.76500000000000001,"5min":0.874,"15min":0.874},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":1.1499999999999999,"5min":1.1499999999999999,"15min":1.1499999999999999},"last":0.59899999999999998}]},{"osd":5,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.63700000000000001},"min":{"1min":0.437,"5min":0.437,"15min":0.437},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.86399999999999999,"5min":0.86399999999999999,"15min":0.86399999999999999},"last":0.61199999999999999}]},{"osd":6,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":1.1850000000000001,"5min":1.1850000000000001,"15min":1.1850000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.65700000000000003,"5min":0.65700000000000003,"15min":0.65700000000000003},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":1.367,"5min":1.367,"15min":1.367},"last":0.624}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63600000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705714,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6884,"kb_used_data":1116,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738268,"statfs":{"total":95995035648,"available":95987986432,"internally_reserved":0,"allocated":1142784,"data_stored":774161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Apr 7 14:07:23 2024","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35399999999999998,"15min":0.34100000000000003},"min":{"1min":0.23300000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.53900000000000003,"5min":0.625,"15min":0.625},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.36099999999999999,"15min":0.34799999999999998},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47599999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.40999999999999998}]},{"osd":2,"last update":"Sun Apr 7 14:06:43 2024","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.38700000000000001,"15min":0.38100000000000001},"min":{"1min":0.28199999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.51000000000000001,"5min":0.51800000000000002,"15min":0.51800000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.434,"5min":0.40200000000000002,"15min":0.39600000000000002},"min":{"1min":0.31,"5min":0.252,"15min":0.252},"max":{"1min":0.61399999999999999,"5min":0.61399999999999999,"15min":0.61399999999999999},"last":0.48799999999999999}]},{"osd":3,"last update":"Sun Apr 7 14:07:14 2024","interfaces":[{"interface":"back","average":{"1min":0.40899999999999997,"5min":0.433,"15min":0.437},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.56000000000000005,"5min":0.59699999999999998,"15min":0.59699999999999998},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42699999999999999,"15min":0.43099999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.52300000000000002,"5min":0.60699999999999998,"15min":0.60699999999999998},"last":0.501}]},{"osd":4,"last update":"Sun Apr 7 14:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.60299999999999998,"15min":0.59899999999999998},"min":{"1min":0.45600000000000002,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.94099999999999995,"5min":0.94099999999999995,"15min":0.94099999999999995},"last":0.94099999999999995},{"interface":"front","average":{"1min":0.63,"5min":0.626,"15min":0.626},"min":{"1min":0.45100000000000001,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":0.95999999999999996,"5min":0.95999999999999996,"15min":0.95999999999999996},"last":0.95999999999999996}]},{"osd":5,"last update":"Sun Apr 7 14:06:56 2024","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.60599999999999998,"15min":0.60599999999999998},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.80600000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.44500000000000001,"5min":0.44500000000000001,"15min":0.44500000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.93000000000000005}]},{"osd":6,"last update":"Sun Apr 7 14:07:19 2024","interfaces":[{"interface":"back","average":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"min":{"1min":0.434,"5min":0.434,"15min":0.434},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"min":{"1min":0.45600000000000002,"5min":0.45600000000000002,"15min":0.45600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.78600000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.78100000000000003}]}]},{"osd":2,"up_from":19,"seq":81604378662,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":6300,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738852,"statfs":{"total":95995035648,"available":95988584448,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.38600000000000001,"15min":0.38},"min":{"1min":0.26600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.246},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.40300000000000002,"15min":0.39900000000000002},"min":{"1min":0.25700000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.59999999999999998,"5min":0.59999999999999998,"15min":0.59999999999999998},"last":0.36399999999999999}]},{"osd":1,"last update":"Sun Apr 7 14:06:41 2024","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.29999999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.41099999999999998,"5min":0.40000000000000002,"15min":0.39800000000000002},"min":{"1min":0.27100000000000002,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.57399999999999995,"5min":0.58399999999999996,"15min":0.58399999999999996},"last":0.25600000000000001}]},{"osd":3,"last update":"Sun Apr 7 14:07:07 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.46100000000000002,"15min":0.46700000000000003},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.315},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.442,"15min":0.44900000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"last":0.33800000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.59499999999999997,"5min":0.60299999999999998,"15min":0.60399999999999998},"min":{"1min":0.38800000000000001,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79100000000000004,"5min":1.04,"15min":1.04},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.628,"5min":0.59199999999999997,"15min":0.58599999999999997},"min":{"1min":0.47099999999999997,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.82399999999999995,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Sun Apr 7 14:06:54 2024","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.376,"5min":0.376,"15min":0.376},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47699999999999998}]},{"osd":6,"last update":"Sun Apr 7 14:07:20 2024","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":1.018,"5min":1.018,"15min":1.018},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.61499999999999999,"15min":0.61499999999999999},"min":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"max":{"1min":0.83699999999999997,"5min":0.83699999999999997,"15min":0.83699999999999997},"last":0.503}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52000000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149730,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5720,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5120,"kb_avail":93739432,"statfs":{"total":95995035648,"available":95989178368,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5242880},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.40799999999999997,"15min":0.41199999999999998},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.59599999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.377,"5min":0.40699999999999997,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.32600000000000001}]},{"osd":1,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.40000000000000002,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.621,"5min":0.621,"15min":0.621},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.40000000000000002,"5min":0.42999999999999999,"15min":0.435},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.33800000000000002}]},{"osd":2,"last update":"Sun Apr 7 14:07:11 2024","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.47399999999999998,"15min":0.48199999999999998},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.315},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.436,"15min":0.44400000000000001},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.66300000000000003,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.27700000000000002}]},{"osd":4,"last update":"Sun Apr 7 14:07:36 2024","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.57299999999999995,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.38,"15min":0.38},"max":{"1min":0.71899999999999997,"5min":0.79600000000000004,"15min":0.79600000000000004},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.59399999999999997,"15min":0.59199999999999997},"min":{"1min":0.45600000000000002,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.88700000000000001,"5min":0.91300000000000003,"15min":0.91300000000000003},"last":0.46800000000000003}]},{"osd":5,"last update":"Sun Apr 7 14:06:57 2024","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.58899999999999997,"15min":0.58899999999999997},"min":{"1min":0.41099999999999998,"5min":0.41099999999999998,"15min":0.41099999999999998},"max":{"1min":0.89200000000000002,"5min":0.89200000000000002,"15min":0.89200000000000002},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.624,"5min":0.624,"15min":0.624},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.88400000000000001,"5min":0.88400000000000001,"15min":0.88400000000000001},"last":0.52500000000000002}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.621,"5min":0.621,"15min":0.621},"min":{"1min":0.45500000000000002,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":0.81899999999999995},"last":0.55500000000000005}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986205,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5784,"kb_used_data":592,"kb_used_omap":0,"kb_used_meta":5184,"kb_avail":93739368,"statfs":{"total":95995035648,"available":95989112832,"internally_reserved":0,"allocated":606208,"data_stored":244968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5308416},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58199999999999996,"15min":0.58699999999999997},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.75600000000000001,"5min":0.75600000000000001,"15min":0.75600000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57999999999999996,"15min":0.57899999999999996},"min":{"1min":0.439,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.753,"5min":0.76600000000000001,"15min":0.76600000000000001},"last":0.48299999999999998}]},{"osd":1,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53300000000000003,"15min":0.52900000000000003},"min":{"1min":0.38200000000000001,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56999999999999995,"15min":0.57699999999999996},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.71099999999999997,"5min":0.80000000000000004,"15min":0.80000000000000004},"last":0.42899999999999999}]},{"osd":2,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.379,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.71299999999999997,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.39600000000000002,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.50900000000000001}]},{"osd":3,"last update":"Sun Apr 7 14:07:33 2024","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.54400000000000004,"15min":0.54300000000000004},"min":{"1min":0.39600000000000002,"5min":0.39100000000000001,"15min":0.39100000000000001},"max":{"1min":0.72399999999999998,"5min":0.76000000000000001,"15min":0.76000000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.56699999999999995,"15min":0.56599999999999995},"min":{"1min":0.41499999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.72099999999999997,"5min":0.78400000000000003,"15min":0.78400000000000003},"last":0.495}]},{"osd":5,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.63,"5min":0.63,"15min":0.63},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.435,"5min":0.435,"15min":0.435},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.376}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.65200000000000002,"5min":0.65200000000000002,"15min":0.65200000000000002},"last":0.56999999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56000000000000005}]}]},{"osd":5,"up_from":39,"seq":167503724567,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":5660,"kb_used_data":596,"kb_used_omap":0,"kb_used_meta":5056,"kb_avail":93739492,"statfs":{"total":95995035648,"available":95989239808,"internally_reserved":0,"allocated":610304,"data_stored":245233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5177344},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61099999999999999}]},{"osd":1,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.54900000000000004}]},{"osd":2,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.60199999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.56200000000000006,"15min":0.56200000000000006},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.63600000000000001}]},{"osd":3,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"min":{"1min":0.34599999999999997,"5min":0.34599999999999997,"15min":0.34599999999999997},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"min":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.56100000000000005}]},{"osd":4,"last update":"Sun Apr 7 14:06:53 2024","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":0.80200000000000005},"last":0.443},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.52900000000000003}]},{"osd":6,"last update":"Sun Apr 7 14:07:18 2024","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.874,"5min":0.874,"15min":0.874},"last":0.51400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-07T14:07:40.754 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-07T14:07:40.754 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-07T14:07:40.754 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-07T14:07:40.754 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph health --format=json 2024-04-07T14:07:40.967 INFO:teuthology.orchestra.run.smithi111.stderr:Inferring config /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/mon.smithi111/config 2024-04-07T14:07:41.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:40 smithi145 ceph-mon[26800]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:41.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:41 smithi111 ceph-mon[24802]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:07:42.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:41 smithi145 ceph-mon[26800]: from='client.14446 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:07:42.652 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:07:42.652 INFO:teuthology.orchestra.run.smithi111.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-07T14:07:42.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:42 smithi111 ceph-mon[24802]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:42.959 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:42 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2545413529' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-07T14:07:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:42 smithi145 ceph-mon[26800]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:42 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2545413529' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-07T14:07:43.285 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-07T14:07:43.285 INFO:tasks.cephadm:Setup complete, yielding 2024-04-07T14:07:43.286 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T14:07:43.295 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:07:43.295 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch status' 2024-04-07T14:07:44.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:44 smithi111 ceph-mon[24802]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:45.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:44 smithi145 ceph-mon[26800]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:45.368 INFO:teuthology.orchestra.run.smithi111.stdout:Backend: cephadm 2024-04-07T14:07:45.368 INFO:teuthology.orchestra.run.smithi111.stdout:Available: Yes 2024-04-07T14:07:45.368 INFO:teuthology.orchestra.run.smithi111.stdout:Paused: No 2024-04-07T14:07:45.931 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:07:45.932 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:07:45.932 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:07:45.932 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:45 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:07:45.955 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ps' 2024-04-07T14:07:46.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:07:46.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:07:46.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:07:46.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:45 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:07:46.912 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:46 smithi111 ceph-mon[24802]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:46.912 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:46 smithi111 ceph-mon[24802]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:47.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:46 smithi145 ceph-mon[26800]: from='client.14454 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:47.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:46 smithi145 ceph-mon[26800]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:47.901 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (4m) 2m ago 5m 15.4M - ba2b418f427c 012d9a12b505 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (5m) 2m ago 5m 6400k - 17.2.0 e1d6a67b021e 0eff85d2a49e 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi145 smithi145 running (5m) 66s ago 5m 6405k - 17.2.0 e1d6a67b021e 9e913164ca86 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (4m) 2m ago 5m 45.6M - 8.3.5 dad864ee21e9 7d5afae8bf58 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.rtrlso smithi111 *:9283 running (6m) 2m ago 6m 448M - 17.2.0 e1d6a67b021e 94ef2bc4f582 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi145.qiunpb smithi145 *:8443,9283 running (4m) 66s ago 4m 412M - 17.2.0 e1d6a67b021e 7da4bd63d481 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (6m) 2m ago 6m 43.9M 2048M 17.2.0 e1d6a67b021e 382c73ff91e2 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi145 smithi145 running (4m) 66s ago 4m 38.8M 2048M 17.2.0 e1d6a67b021e c8daa5dcac8e 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (5m) 2m ago 5m 15.2M - 1dbe0e931976 ba6021f0730f 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi145 smithi145 *:9100 running (4m) 66s ago 4m 19.7M - 1dbe0e931976 bf68fb6f0950 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (4m) 2m ago 4m 32.2M 3216M 17.2.0 e1d6a67b021e ea5afbacec5e 2024-04-07T14:07:47.902 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (3m) 2m ago 3m 36.9M 3216M 17.2.0 e1d6a67b021e 1da0bd36d060 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (3m) 2m ago 3m 30.4M 3216M 17.2.0 e1d6a67b021e c6ab9df37c92 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi111 running (2m) 2m ago 2m 10.9M 3216M 17.2.0 e1d6a67b021e 195f5ab7aba6 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi145 running (2m) 66s ago 2m 32.9M 3984M 17.2.0 e1d6a67b021e 06ffbb78268c 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi145 running (2m) 66s ago 2m 36.3M 3984M 17.2.0 e1d6a67b021e 5ffb10ce0799 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 smithi145 running (96s) 66s ago 95s 30.9M 3984M 17.2.0 e1d6a67b021e 5e283a14fbad 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 smithi145 running (71s) 66s ago 70s 13.2M 3984M 17.2.0 e1d6a67b021e d5503daa3f47 2024-04-07T14:07:47.903 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (4m) 2m ago 4m 49.5M - 514e6a882f6e 1965f3714193 2024-04-07T14:07:48.528 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch ls' 2024-04-07T14:07:48.839 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:48 smithi111 ceph-mon[24802]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:49.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:48 smithi145 ceph-mon[26800]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:49.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:49 smithi111 ceph-mon[24802]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:50.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:49 smithi145 ceph-mon[26800]: from='client.14458 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:50.534 INFO:teuthology.orchestra.run.smithi111.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager ?:9093,9094 1/1 2m ago 6m count:1 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:crash 2/2 2m ago 6m * 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:grafana ?:3000 1/1 2m ago 6m count:1 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:mgr 2/2 2m ago 6m count:2 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:mon 2/2 2m ago 5m smithi111:172.21.15.111=smithi111;smithi145:172.21.15.145=smithi145;count:2 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter ?:9100 2/2 2m ago 6m * 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:osd 8 2m ago - 2024-04-07T14:07:50.535 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus ?:9095 1/1 2m ago 6m count:1 2024-04-07T14:07:50.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:50 smithi111 ceph-mon[24802]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:51.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:50 smithi145 ceph-mon[26800]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:51.199 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch host ls' 2024-04-07T14:07:51.752 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:51 smithi111 ceph-mon[24802]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:52.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:51 smithi145 ceph-mon[26800]: from='client.14462 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:52.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:52 smithi111 ceph-mon[24802]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:53.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:52 smithi145 ceph-mon[26800]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:HOST ADDR LABELS STATUS 2024-04-07T14:07:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 172.21.15.111 2024-04-07T14:07:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:smithi145 172.21.15.145 2024-04-07T14:07:53.104 INFO:teuthology.orchestra.run.smithi111.stdout:2 hosts in cluster 2024-04-07T14:07:53.835 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph orch device ls' 2024-04-07T14:07:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:54 smithi111 ceph-mon[24802]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:54.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:54 smithi111 ceph-mon[24802]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:54 smithi145 ceph-mon[26800]: from='client.14466 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:54 smithi145 ceph-mon[26800]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:55.869 INFO:teuthology.orchestra.run.smithi111.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REJECT REASONS 2024-04-07T14:07:55.869 INFO:teuthology.orchestra.run.smithi111.stdout:smithi111 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VZ400BGN 400G LVM detected, locked 2024-04-07T14:07:55.870 INFO:teuthology.orchestra.run.smithi111.stdout:smithi145 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N9400BGN 400G LVM detected, locked 2024-04-07T14:07:56.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:56 smithi111 ceph-mon[24802]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:56.516 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-07T14:07:56.527 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:07:56.528 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-07T14:07:56.555 INFO:teuthology.orchestra.run.smithi111.stderr:+ systemctl stop nfs-server 2024-04-07T14:07:56.562 INFO:tasks.vip:Running commands on role host.b host ubuntu@smithi145.front.sepia.ceph.com 2024-04-07T14:07:56.563 DEBUG:teuthology.orchestra.run.smithi145:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'systemctl stop nfs-server' 2024-04-07T14:07:56.594 INFO:teuthology.orchestra.run.smithi145.stderr:+ systemctl stop nfs-server 2024-04-07T14:07:56.603 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T14:07:56.611 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:07:56.611 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph fs volume create foofs' 2024-04-07T14:07:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:56 smithi145 ceph-mon[26800]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:57.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:57 smithi111 ceph-mon[24802]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:57.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:57 smithi145 ceph-mon[26800]: from='client.14470 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:58.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:58 smithi111 ceph-mon[24802]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:58.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:58 smithi145 ceph-mon[26800]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:07:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:59 smithi111 ceph-mon[24802]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:59.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:07:59 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-07T14:07:59.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:59 smithi145 ceph-mon[26800]: from='client.14474 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "foofs", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:07:59.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:07:59 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]: dispatch 2024-04-07T14:08:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-07T14:08:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:00 smithi111 ceph-mon[24802]: osdmap e60: 8 total, 8 up, 8 in 2024-04-07T14:08:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:00 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-07T14:08:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:00 smithi111 ceph-mon[24802]: pgmap v209: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:00.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:00 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111[24798]: 2024-04-07T14:08:00.409+0000 7f412bf72700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-07T14:08:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.meta"}]': finished 2024-04-07T14:08:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:00 smithi145 ceph-mon[26800]: osdmap e60: 8 total, 8 up, 8 in 2024-04-07T14:08:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:00 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]: dispatch 2024-04-07T14:08:00.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:00 smithi145 ceph-mon[26800]: pgmap v209: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:01.068 INFO:teuthology.run_tasks:Running task cephadm.wait_for_service... 2024-04-07T14:08:01.085 INFO:tasks.cephadm:Waiting for ceph service mds.foofs to start (timeout 300)... 2024-04-07T14:08:01.085 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-07T14:08:01.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-07T14:08:01.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: osdmap e61: 8 total, 8 up, 8 in 2024-04-07T14:08:01.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-07T14:08:01.632 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: osdmap e62: 8 total, 8 up, 8 in 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: fsmap foofs:0 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: Saving service mds.foofs spec with placement count:2 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi111.pvxsdw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi111.pvxsdw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:01.633 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:01 smithi111 ceph-mon[24802]: Deploying daemon mds.foofs.smithi111.pvxsdw on smithi111 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": "cephfs.foofs.data"}]': finished 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: osdmap e61: 8 total, 8 up, 8 in 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]: dispatch 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: osdmap e62: 8 total, 8 up, 8 in 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "fs new", "fs_name": "foofs", "metadata": "cephfs.foofs.meta", "data": "cephfs.foofs.data"}]': finished 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: fsmap foofs:0 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: Saving service mds.foofs spec with placement count:2 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.675 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi111.pvxsdw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi111.pvxsdw", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:01.676 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:01 smithi145 ceph-mon[26800]: Deploying daemon mds.foofs.smithi111.pvxsdw on smithi111 2024-04-07T14:08:02.722 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:02 smithi111 ceph-mon[24802]: osdmap e63: 8 total, 8 up, 8 in 2024-04-07T14:08:02.722 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:02 smithi111 ceph-mon[24802]: pgmap v213: 65 pgs: 6 creating+peering, 1 active+clean, 58 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:02 smithi145 ceph-mon[26800]: osdmap e63: 8 total, 8 up, 8 in 2024-04-07T14:08:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:02 smithi145 ceph-mon[26800]: pgmap v213: 65 pgs: 6 creating+peering, 1 active+clean, 58 unknown; 577 KiB data, 48 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:03.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:03 smithi111 ceph-mon[24802]: osdmap e64: 8 total, 8 up, 8 in 2024-04-07T14:08:03.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:03 smithi145 ceph-mon[26800]: osdmap e64: 8 total, 8 up, 8 in 2024-04-07T14:08:04.320 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:08:04.320 INFO:teuthology.orchestra.run.smithi111.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-07T14:01:37.174087Z", "last_refresh": "2024-04-07T14:05:06.881241Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-07T14:01:30.819586Z", "last_refresh": "2024-04-07T14:05:06.881303Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-07T14:01:33.861220Z", "last_refresh": "2024-04-07T14:05:06.881363Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-07T14:08:00.428605Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-07T14:08:00.423992Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-07T14:01:28.812988Z", "last_refresh": "2024-04-07T14:05:06.881172Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi111:172.21.15.111=smithi111", "smithi145:172.21.15.145=smithi145"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-07T14:02:20.053692Z", "last_refresh": "2024-04-07T14:05:06.881016Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-07T14:01:35.085029Z", "last_refresh": "2024-04-07T14:05:06.881423Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-07T14:05:06.881541Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-07T14:01:32.087081Z", "last_refresh": "2024-04-07T14:05:06.881482Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-07T14:08:04.574 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:04 smithi111 ceph-mon[24802]: pgmap v215: 65 pgs: 6 creating+peering, 36 active+clean, 23 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:04.761 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:04 smithi145 ceph-mon[26800]: pgmap v215: 65 pgs: 6 creating+peering, 36 active+clean, 23 unknown; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:05.116 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-07T14:08:05.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:05.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:05.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi145.fczrwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-07T14:08:05.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi145.fczrwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-07T14:08:05.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: Deploying daemon mds.foofs.smithi145.fczrwb on smithi145 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: daemon mds.foofs.smithi111.pvxsdw assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: Cluster is now healthy 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: mds.? [v2:172.21.15.111:6834/606651965,v1:172.21.15.111:6835/606651965] up:boot 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:creating} 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi111.pvxsdw"}]: dispatch 2024-04-07T14:08:05.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:05 smithi111 ceph-mon[24802]: daemon mds.foofs.smithi111.pvxsdw is now active in filesystem foofs as rank 0 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='client.14478 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi145.fczrwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "mds.foofs.smithi145.fczrwb", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: Deploying daemon mds.foofs.smithi145.fczrwb on smithi145 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: daemon mds.foofs.smithi111.pvxsdw assigned to filesystem foofs as rank 0 (now has 1 ranks) 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-07T14:08:06.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-07T14:08:06.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: Cluster is now healthy 2024-04-07T14:08:06.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: mds.? [v2:172.21.15.111:6834/606651965,v1:172.21.15.111:6835/606651965] up:boot 2024-04-07T14:08:06.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:creating} 2024-04-07T14:08:06.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi111.pvxsdw"}]: dispatch 2024-04-07T14:08:06.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:05 smithi145 ceph-mon[26800]: daemon mds.foofs.smithi111.pvxsdw is now active in filesystem foofs as rank 0 2024-04-07T14:08:06.118 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-07T14:08:06.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:06 smithi145 ceph-mon[26800]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:06.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:06 smithi145 ceph-mon[26800]: mds.? [v2:172.21.15.111:6834/606651965,v1:172.21.15.111:6835/606651965] up:active 2024-04-07T14:08:06.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:06 smithi145 ceph-mon[26800]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 2024-04-07T14:08:06.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:06 smithi111 ceph-mon[24802]: pgmap v216: 65 pgs: 65 active+clean; 577 KiB data, 49 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:06.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:06 smithi111 ceph-mon[24802]: mds.? [v2:172.21.15.111:6834/606651965,v1:172.21.15.111:6835/606651965] up:active 2024-04-07T14:08:06.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:06 smithi111 ceph-mon[24802]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 2024-04-07T14:08:08.120 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:08:08.120 INFO:teuthology.orchestra.run.smithi111.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-07T14:01:37.174087Z", "last_refresh": "2024-04-07T14:05:06.881241Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-07T14:01:30.819586Z", "last_refresh": "2024-04-07T14:05:06.881303Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-07T14:01:33.861220Z", "last_refresh": "2024-04-07T14:05:06.881363Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-07T14:08:00.428605Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-07T14:08:00.423992Z", "running": 0, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-07T14:01:28.812988Z", "last_refresh": "2024-04-07T14:05:06.881172Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi111:172.21.15.111=smithi111", "smithi145:172.21.15.145=smithi145"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-07T14:02:20.053692Z", "last_refresh": "2024-04-07T14:05:06.881016Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-07T14:01:35.085029Z", "last_refresh": "2024-04-07T14:05:06.881423Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-07T14:05:06.881541Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-07T14:01:32.087081Z", "last_refresh": "2024-04-07T14:05:06.881482Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:08.656 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: mds.? [v2:172.21.15.145:6832/834543199,v1:172.21.15.145:6833/834543199] up:boot 2024-04-07T14:08:08.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 1 up:standby 2024-04-07T14:08:08.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi145.fczrwb"}]: dispatch 2024-04-07T14:08:08.657 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:08 smithi145 ceph-mon[26800]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 1 up:standby 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: pgmap v217: 65 pgs: 65 active+clean; 577 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 0 B/s wr, 2 op/s 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: mds.? [v2:172.21.15.145:6832/834543199,v1:172.21.15.145:6833/834543199] up:boot 2024-04-07T14:08:08.661 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 1 up:standby 2024-04-07T14:08:08.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi145.fczrwb"}]: dispatch 2024-04-07T14:08:08.662 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:08 smithi111 ceph-mon[24802]: fsmap foofs:1 {0=foofs.smithi111.pvxsdw=up:active} 1 up:standby 2024-04-07T14:08:09.251 INFO:tasks.cephadm:mds.foofs has 0/2 2024-04-07T14:08:09.561 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:09 smithi111 ceph-mon[24802]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:09.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:09 smithi145 ceph-mon[26800]: from='client.14486 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:10.252 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-07T14:08:10.541 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:10 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:10.541 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:10 smithi111 ceph-mon[24802]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-07T14:08:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:10 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:10.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:10 smithi145 ceph-mon[26800]: pgmap v218: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-07T14:08:12.772 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:12 smithi111 ceph-mon[24802]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-07T14:08:12.772 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:12 smithi145 ceph-mon[26800]: pgmap v219: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-04-07T14:08:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:13.070 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:08:13.070 INFO:teuthology.orchestra.run.smithi111.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-07T14:01:37.174087Z", "last_refresh": "2024-04-07T14:05:06.881241Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-07T14:01:30.819586Z", "last_refresh": "2024-04-07T14:05:06.881303Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-07T14:01:33.861220Z", "last_refresh": "2024-04-07T14:05:06.881363Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-07T14:08:00.428605Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-07T14:08:00.423992Z", "last_refresh": "2024-04-07T14:08:09.320285Z", "running": 1, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-07T14:01:28.812988Z", "last_refresh": "2024-04-07T14:05:06.881172Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi111:172.21.15.111=smithi111", "smithi145:172.21.15.145=smithi145"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-07T14:02:20.053692Z", "last_refresh": "2024-04-07T14:05:06.881016Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-07T14:01:35.085029Z", "last_refresh": "2024-04-07T14:05:06.881423Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-07T14:05:06.881541Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-07T14:01:32.087081Z", "last_refresh": "2024-04-07T14:05:06.881482Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-07T14:08:14.177 INFO:tasks.cephadm:mds.foofs has 1/2 2024-04-07T14:08:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-07T14:08:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:14.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:14.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:14.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:14.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:14 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='client.14490 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: pgmap v220: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:14 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:15.178 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- ceph orch ls -f json 2024-04-07T14:08:16.669 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:16 smithi111 ceph-mon[24802]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-07T14:08:16.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:16 smithi145 ceph-mon[26800]: pgmap v221: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-07T14:08:17.227 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:08:17.227 INFO:teuthology.orchestra.run.smithi111.stdout:[{"placement": {"count": 1}, "service_name": "alertmanager", "service_type": "alertmanager", "status": {"created": "2024-04-07T14:01:37.174087Z", "last_refresh": "2024-04-07T14:08:14.342471Z", "ports": [9093, 9094], "running": 1, "size": 1}}, {"placement": {"host_pattern": "*"}, "service_name": "crash", "service_type": "crash", "status": {"created": "2024-04-07T14:01:30.819586Z", "last_refresh": "2024-04-07T14:08:09.319651Z", "running": 2, "size": 2}}, {"placement": {"count": 1}, "service_name": "grafana", "service_type": "grafana", "status": {"created": "2024-04-07T14:01:33.861220Z", "last_refresh": "2024-04-07T14:08:14.342593Z", "ports": [3000], "running": 1, "size": 1}}, {"events": ["2024-04-07T14:08:00.428605Z service:mds.foofs [INFO] \"service was created\""], "placement": {"count": 2}, "service_id": "foofs", "service_name": "mds.foofs", "service_type": "mds", "status": {"created": "2024-04-07T14:08:00.423992Z", "last_refresh": "2024-04-07T14:08:09.320285Z", "running": 2, "size": 2}}, {"placement": {"count": 2}, "service_name": "mgr", "service_type": "mgr", "status": {"created": "2024-04-07T14:01:28.812988Z", "last_refresh": "2024-04-07T14:08:09.319803Z", "running": 2, "size": 2}}, {"placement": {"count": 2, "hosts": ["smithi111:172.21.15.111=smithi111", "smithi145:172.21.15.145=smithi145"]}, "service_name": "mon", "service_type": "mon", "status": {"created": "2024-04-07T14:02:20.053692Z", "last_refresh": "2024-04-07T14:08:09.319894Z", "running": 2, "size": 2}}, {"placement": {"host_pattern": "*"}, "service_name": "node-exporter", "service_type": "node-exporter", "status": {"created": "2024-04-07T14:01:35.085029Z", "last_refresh": "2024-04-07T14:08:09.319963Z", "ports": [9100], "running": 2, "size": 2}}, {"service_name": "osd", "service_type": "osd", "spec": {"filter_logic": "AND", "objectstore": "bluestore"}, "status": {"container_image_id": "e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9", "container_image_name": "quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a", "last_refresh": "2024-04-07T14:08:09.320028Z", "running": 8, "size": 8}, "unmanaged": true}, {"placement": {"count": 1}, "service_name": "prometheus", "service_type": "prometheus", "status": {"created": "2024-04-07T14:01:32.087081Z", "last_refresh": "2024-04-07T14:08:14.342713Z", "ports": [9095], "running": 1, "size": 1}}] 2024-04-07T14:08:18.017 INFO:tasks.cephadm:mds.foofs has 2/2 2024-04-07T14:08:18.017 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-07T14:08:18.026 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:08:18.026 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs cluster create foo --placement=2 || ceph nfs cluster create cephfs foo --placement=2' 2024-04-07T14:08:18.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:18 smithi145 ceph-mon[26800]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:18.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:18 smithi145 ceph-mon[26800]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-07T14:08:18.924 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:18 smithi111 ceph-mon[24802]: from='client.14494 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-07T14:08:18.925 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:18 smithi111 ceph-mon[24802]: pgmap v222: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 3 op/s 2024-04-07T14:08:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:20 smithi111 ceph-mon[24802]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-07T14:08:20.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:20 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-07T14:08:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:20 smithi145 ceph-mon[26800]: pgmap v223: 65 pgs: 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s wr, 2 op/s 2024-04-07T14:08:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:20 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool create", "pool": ".nfs"}]: dispatch 2024-04-07T14:08:21.679 INFO:teuthology.orchestra.run.smithi111.stdout:NFS Cluster Created Successfully 2024-04-07T14:08:21.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:21 smithi111 ceph-mon[24802]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:21.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:21 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-07T14:08:21.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:21 smithi111 ceph-mon[24802]: osdmap e65: 8 total, 8 up, 8 in 2024-04-07T14:08:21.977 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:21 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-07T14:08:22.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:21 smithi145 ceph-mon[26800]: from='client.14498 -' entity='client.admin' cmd=[{"prefix": "nfs cluster create", "cluster_id": "foo", "placement": "2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:22.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:21 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool create", "pool": ".nfs"}]': finished 2024-04-07T14:08:22.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:21 smithi145 ceph-mon[26800]: osdmap e65: 8 total, 8 up, 8 in 2024-04-07T14:08:22.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:21 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]: dispatch 2024-04-07T14:08:22.404 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'ceph nfs export create cephfs --fsname foofs --clusterid foo --binding /fake || ceph nfs export create cephfs --fsname foofs --cluster-id foo --pseudo-path /fake' 2024-04-07T14:08:22.909 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: pgmap v225: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:22.909 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: osdmap e66: 8 total, 8 up, 8 in 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: Saving service nfs.foo spec with placement count:2 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: Creating key for client.nfs.foo.0.0.smithi111.jegwqs 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-07T14:08:22.910 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T14:08:22.911 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:22 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: pgmap v225: 97 pgs: 32 unknown, 65 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail 2024-04-07T14:08:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pool application enable", "pool": ".nfs", "app": "nfs"}]': finished 2024-04-07T14:08:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: osdmap e66: 8 total, 8 up, 8 in 2024-04-07T14:08:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: Saving service nfs.foo spec with placement count:2 2024-04-07T14:08:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: Creating key for client.nfs.foo.0.0.smithi111.jegwqs 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: Ensuring nfs.foo.0 is in the ganesha grace table 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-07T14:08:23.024 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-07T14:08:23.025 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T14:08:23.025 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.0.0.smithi111.jegwqs-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T14:08:23.025 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:22 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:23.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:23 smithi111 ceph-mon[24802]: Rados config object exists: conf-nfs.foo 2024-04-07T14:08:23.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:23 smithi111 ceph-mon[24802]: Creating key for client.nfs.foo.0.0.smithi111.jegwqs-rgw 2024-04-07T14:08:23.956 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:23 smithi111 ceph-mon[24802]: Deploying daemon nfs.foo.0.0.smithi111.jegwqs on smithi111 2024-04-07T14:08:23.957 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:23 smithi111 ceph-mon[24802]: osdmap e67: 8 total, 8 up, 8 in 2024-04-07T14:08:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:23 smithi145 ceph-mon[26800]: Rados config object exists: conf-nfs.foo 2024-04-07T14:08:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:23 smithi145 ceph-mon[26800]: Creating key for client.nfs.foo.0.0.smithi111.jegwqs-rgw 2024-04-07T14:08:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:23 smithi145 ceph-mon[26800]: Deploying daemon nfs.foo.0.0.smithi111.jegwqs on smithi111 2024-04-07T14:08:24.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:23 smithi145 ceph-mon[26800]: osdmap e67: 8 total, 8 up, 8 in 2024-04-07T14:08:24.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:24 smithi111 ceph-mon[24802]: pgmap v228: 97 pgs: 1 creating+activating, 10 unknown, 86 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-07T14:08:24.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:24 smithi111 ceph-mon[24802]: mgrmap e20: smithi111.rtrlso(active, since 5m), standbys: smithi145.qiunpb 2024-04-07T14:08:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:24 smithi145 ceph-mon[26800]: pgmap v228: 97 pgs: 1 creating+activating, 10 unknown, 86 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-07T14:08:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:24 smithi145 ceph-mon[26800]: mgrmap e20: smithi111.rtrlso(active, since 5m), standbys: smithi145.qiunpb 2024-04-07T14:08:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:Invalid command: Unexpected argument '--clusterid' 2024-04-07T14:08:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:nfs export create cephfs [] [--readonly] [--client_addr ...] [--squash ] : Create a CephFS export 2024-04-07T14:08:25.787 INFO:teuthology.orchestra.run.smithi111.stderr:Error EINVAL: invalid command 2024-04-07T14:08:26.150 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:08:26.150 INFO:teuthology.orchestra.run.smithi111.stdout: "bind": "/fake", 2024-04-07T14:08:26.150 INFO:teuthology.orchestra.run.smithi111.stdout: "fs": "foofs", 2024-04-07T14:08:26.150 INFO:teuthology.orchestra.run.smithi111.stdout: "path": "/", 2024-04-07T14:08:26.151 INFO:teuthology.orchestra.run.smithi111.stdout: "cluster": "foo", 2024-04-07T14:08:26.151 INFO:teuthology.orchestra.run.smithi111.stdout: "mode": "RW" 2024-04-07T14:08:26.151 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: pgmap v229: 97 pgs: 1 creating+activating, 96 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-07T14:08:26.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-07T14:08:26.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:26.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-07T14:08:26.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:26 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: pgmap v229: 97 pgs: 1 creating+activating, 96 active+clean; 579 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 341 B/s wr, 0 op/s 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]: dispatch 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo"]}]': finished 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]: dispatch 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.mgr.nfs.grace.nfs.foo", "caps": ["mon", "allow r", "osd", "allow rwx pool .nfs"]}]': finished 2024-04-07T14:08:26.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:26.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]: dispatch 2024-04-07T14:08:26.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:26 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1", "caps": ["mon", "allow r", "osd", "allow rw pool=.nfs namespace=foo, allow rw tag cephfs data=foofs", "mds", "allow rw path=/"], "format": "json"}]': finished 2024-04-07T14:08:26.870 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -- bash -c 'while ! ceph orch ls | grep nfs | grep 2/2 ; do sleep 1 ; done' 2024-04-07T14:08:27.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:27 smithi111 ceph-mon[24802]: Creating key for client.nfs.foo.1.0.smithi145.lsscmc 2024-04-07T14:08:27.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:27 smithi111 ceph-mon[24802]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-07T14:08:27.476 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:27 smithi111 ceph-mon[24802]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:27.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:27 smithi145 ceph-mon[26800]: Creating key for client.nfs.foo.1.0.smithi145.lsscmc 2024-04-07T14:08:27.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:27 smithi145 ceph-mon[26800]: Ensuring nfs.foo.1 is in the ganesha grace table 2024-04-07T14:08:27.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:27 smithi145 ceph-mon[26800]: from='client.14520 -' entity='client.admin' cmd=[{"prefix": "nfs export create cephfs", "fsname": "foofs", "cluster_id": "foo", "pseudo_path": "/fake", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:28.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:28 smithi111 ceph-mon[24802]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-07T14:08:28.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:28 smithi145 ceph-mon[26800]: pgmap v230: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.6 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-07T14:08:29.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:29.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-07T14:08:29.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-07T14:08:29.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T14:08:29.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T14:08:29.727 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:29 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='client.14536 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]: dispatch 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth rm", "entity": "client.mgr.nfs.grace.nfs.foo"}]': finished 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "auth get-or-create", "entity": "client.nfs.foo.1.0.smithi145.lsscmc-rgw", "caps": ["mon", "allow r", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-04-07T14:08:29.735 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:29 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:30.680 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:30 smithi145 ceph-mon[26800]: Rados config object exists: conf-nfs.foo 2024-04-07T14:08:30.681 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:30 smithi145 ceph-mon[26800]: Creating key for client.nfs.foo.1.0.smithi145.lsscmc-rgw 2024-04-07T14:08:30.681 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:30 smithi145 ceph-mon[26800]: Deploying daemon nfs.foo.1.0.smithi145.lsscmc on smithi145 2024-04-07T14:08:30.681 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:30 smithi145 ceph-mon[26800]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-07T14:08:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:30 smithi111 ceph-mon[24802]: Rados config object exists: conf-nfs.foo 2024-04-07T14:08:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:30 smithi111 ceph-mon[24802]: Creating key for client.nfs.foo.1.0.smithi145.lsscmc-rgw 2024-04-07T14:08:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:30 smithi111 ceph-mon[24802]: Deploying daemon nfs.foo.1.0.smithi145.lsscmc on smithi145 2024-04-07T14:08:30.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:30 smithi111 ceph-mon[24802]: pgmap v231: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 3 op/s 2024-04-07T14:08:31.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:31 smithi111 ceph-mon[24802]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:31.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:31 smithi145 ceph-mon[26800]: from='client.14550 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:32.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:32 smithi145 ceph-mon[26800]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-07T14:08:32.822 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: from='client.14554 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:32.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:32.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:32.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:32.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:32.830 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:32 smithi111 ceph-mon[24802]: pgmap v232: 97 pgs: 97 active+clean; 579 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1.1 KiB/s wr, 2 op/s 2024-04-07T14:08:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:33 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:33.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:33 smithi111 ceph-mon[24802]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:33.978 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:33 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:33.978 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:33 smithi145 ceph-mon[26800]: from='client.14570 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:34.957 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:34 smithi111 ceph-mon[24802]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-07T14:08:34.957 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:35.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:34 smithi145 ceph-mon[26800]: pgmap v233: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.8 KiB/s rd, 2.3 KiB/s wr, 6 op/s 2024-04-07T14:08:35.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:35 smithi145 ceph-mon[26800]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:35.877 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:35 smithi111 ceph-mon[24802]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:36.569 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo ?:2049 2/2 2s ago 14s count:2 2024-04-07T14:08:36.925 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:36.925 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:36.926 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:36 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='client.14578 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: pgmap v234: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:36 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:08:37.379 INFO:teuthology.run_tasks:Running task vip.exec... 2024-04-07T14:08:37.389 INFO:tasks.vip:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:08:37.389 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'mkdir /mnt/foo' 2024-04-07T14:08:37.419 INFO:teuthology.orchestra.run.smithi111.stderr:+ mkdir /mnt/foo 2024-04-07T14:08:37.434 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'while ! mount -t nfs $(hostname):/fake /mnt/foo -o sync ; do sleep 5 ; done' 2024-04-07T14:08:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:++ hostname 2024-04-07T14:08:37.502 INFO:teuthology.orchestra.run.smithi111.stderr:+ mount -t nfs smithi111:/fake /mnt/foo -o sync 2024-04-07T14:08:37.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:08:37.891 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:08:37.918 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c 'echo test > /mnt/foo/testfile' 2024-04-07T14:08:37.984 INFO:teuthology.orchestra.run.smithi111.stderr:+ echo test 2024-04-07T14:08:37.994 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -ex -c sync 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='client.14582 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:08:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:08:38.059 INFO:teuthology.orchestra.run.smithi111.stderr:+ sync 2024-04-07T14:08:38.330 INFO:teuthology.run_tasks:Running task parallel... 2024-04-07T14:08:38.351 INFO:teuthology.task.parallel:starting parallel... 2024-04-07T14:08:38.352 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-07T14:08:38.359 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-07T14:08:38.360 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:08:38.360 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-04-07T14:08:38.360 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-04-07T14:08:38.360 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-07T14:08:38.361 INFO:teuthology.task.exec:Executing custom commands... 2024-04-07T14:08:38.361 INFO:teuthology.task.exec:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:08:38.361 DEBUG:teuthology.orchestra.run.smithi111:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'cd /mnt/foo && dbench 5 -t 600 || true' 2024-04-07T14:08:38.407 INFO:teuthology.orchestra.run.smithi111.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-04-07T14:08:38.407 INFO:teuthology.orchestra.run.smithi111.stdout: 2024-04-07T14:08:38.407 INFO:teuthology.orchestra.run.smithi111.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-04-07T14:08:38.408 INFO:teuthology.orchestra.run.smithi111.stdout:3 of 5 processes prepared for launch 0 sec 2024-04-07T14:08:38.508 INFO:teuthology.orchestra.run.smithi111.stdout:5 of 5 processes prepared for launch 0 sec 2024-04-07T14:08:38.508 INFO:teuthology.orchestra.run.smithi111.stdout:releasing clients 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-04-07T14:08:38.836 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:38 smithi111 ceph-mon[24802]: osdmap e68: 8 total, 8 up, 8 in 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: pgmap v235: 97 pgs: 97 active+clean; 581 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 2.1 KiB/s wr, 6 op/s 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [7, 2]}]': finished 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [1, 5]}]': finished 2024-04-07T14:08:39.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:38 smithi145 ceph-mon[26800]: osdmap e68: 8 total, 8 up, 8 in 2024-04-07T14:08:39.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 189 37.82 MB/sec warmup 1 sec latency 26.506 ms 2024-04-07T14:08:39.842 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:39 smithi111 ceph-mon[24802]: osdmap e69: 8 total, 8 up, 8 in 2024-04-07T14:08:40.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:39 smithi145 ceph-mon[26800]: osdmap e69: 8 total, 8 up, 8 in 2024-04-07T14:08:40.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 332 30.00 MB/sec warmup 2 sec latency 31.334 ms 2024-04-07T14:08:40.881 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:40 smithi111 ceph-mon[24802]: pgmap v238: 97 pgs: 97 active+clean; 6.9 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 813 KiB/s wr, 20 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:41.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:40 smithi145 ceph-mon[26800]: pgmap v238: 97 pgs: 97 active+clean; 6.9 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 4.0 KiB/s rd, 813 KiB/s wr, 20 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:41.509 INFO:teuthology.orchestra.run.smithi111.stdout: 5 493 28.58 MB/sec warmup 3 sec latency 23.964 ms 2024-04-07T14:08:41.822 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-04-07T14:08:42.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 726 26.61 MB/sec warmup 4 sec latency 31.266 ms 2024-04-07T14:08:42.800 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:42 smithi111 ceph-mon[24802]: pgmap v239: 97 pgs: 97 active+clean; 9.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s wr, 19 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:42 smithi145 ceph-mon[26800]: pgmap v239: 97 pgs: 97 active+clean; 9.6 MiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.1 MiB/s wr, 19 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:43.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 1184 23.02 MB/sec warmup 5 sec latency 22.539 ms 2024-04-07T14:08:44.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 2268 22.67 MB/sec warmup 6 sec latency 26.029 ms 2024-04-07T14:08:44.819 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:44 smithi111 ceph-mon[24802]: pgmap v240: 97 pgs: 97 active+clean; 66 MiB data, 135 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 8.7 MiB/s wr, 449 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:45.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:44 smithi145 ceph-mon[26800]: pgmap v240: 97 pgs: 97 active+clean; 66 MiB data, 135 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 8.7 MiB/s wr, 449 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:45.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 3093 24.31 MB/sec warmup 7 sec latency 20.644 ms 2024-04-07T14:08:46.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 3837 24.89 MB/sec warmup 8 sec latency 20.134 ms 2024-04-07T14:08:46.582 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:46 smithi111 ceph-mon[24802]: pgmap v241: 97 pgs: 97 active+clean; 109 MiB data, 163 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 15 MiB/s wr, 612 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:46.607 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph config set global log_to_journald false --force' 2024-04-07T14:08:46.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:46 smithi145 ceph-mon[26800]: pgmap v241: 97 pgs: 97 active+clean; 109 MiB data, 163 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 15 MiB/s wr, 612 op/s; 69 B/s, 0 objects/s recovering 2024-04-07T14:08:47.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 4496 22.77 MB/sec warmup 9 sec latency 21.641 ms 2024-04-07T14:08:48.511 INFO:teuthology.orchestra.run.smithi111.stdout: 5 5465 22.78 MB/sec warmup 10 sec latency 21.888 ms 2024-04-07T14:08:48.847 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:48 smithi111 ceph-mon[24802]: pgmap v242: 97 pgs: 97 active+clean; 114 MiB data, 181 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 13 MiB/s wr, 587 op/s; 55 B/s, 0 objects/s recovering 2024-04-07T14:08:49.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:48 smithi145 ceph-mon[26800]: pgmap v242: 97 pgs: 97 active+clean; 114 MiB data, 181 MiB used, 715 GiB / 715 GiB avail; 2.3 KiB/s rd, 13 MiB/s wr, 587 op/s; 55 B/s, 0 objects/s recovering 2024-04-07T14:08:49.510 INFO:teuthology.orchestra.run.smithi111.stdout: 5 6144 22.04 MB/sec warmup 11 sec latency 49.887 ms 2024-04-07T14:08:50.516 INFO:teuthology.orchestra.run.smithi111.stdout: 5 7062 23.50 MB/sec warmup 12 sec latency 23.595 ms 2024-04-07T14:08:50.890 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:50 smithi111 ceph-mon[24802]: pgmap v243: 97 pgs: 97 active+clean; 132 MiB data, 268 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 15 MiB/s wr, 885 op/s; 50 B/s, 0 objects/s recovering 2024-04-07T14:08:51.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:50 smithi145 ceph-mon[26800]: pgmap v243: 97 pgs: 97 active+clean; 132 MiB data, 268 MiB used, 715 GiB / 715 GiB avail; 2.1 KiB/s rd, 15 MiB/s wr, 885 op/s; 50 B/s, 0 objects/s recovering 2024-04-07T14:08:51.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 7756 22.84 MB/sec warmup 13 sec latency 22.276 ms 2024-04-07T14:08:52.489 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph mgr module enable nfs --force' 2024-04-07T14:08:52.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 8316 22.05 MB/sec warmup 14 sec latency 42.551 ms 2024-04-07T14:08:52.652 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:52 smithi111 ceph-mon[24802]: pgmap v244: 97 pgs: 97 active+clean; 132 MiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 14 MiB/s wr, 816 op/s 2024-04-07T14:08:53.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:52 smithi145 ceph-mon[26800]: pgmap v244: 97 pgs: 97 active+clean; 132 MiB data, 273 MiB used, 715 GiB / 715 GiB avail; 1.9 KiB/s rd, 14 MiB/s wr, 816 op/s 2024-04-07T14:08:53.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 9003 21.54 MB/sec warmup 15 sec latency 94.171 ms 2024-04-07T14:08:54.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 9590 20.98 MB/sec warmup 16 sec latency 53.224 ms 2024-04-07T14:08:54.718 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:54 smithi111 ceph-mon[24802]: pgmap v245: 97 pgs: 97 active+clean; 140 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 16 MiB/s wr, 1.03k op/s 2024-04-07T14:08:55.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:54 smithi145 ceph-mon[26800]: pgmap v245: 97 pgs: 97 active+clean; 140 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 8.1 KiB/s rd, 16 MiB/s wr, 1.03k op/s 2024-04-07T14:08:55.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 10159 21.12 MB/sec warmup 17 sec latency 33.413 ms 2024-04-07T14:08:56.484 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:56 smithi111 ceph-mon[24802]: pgmap v246: 97 pgs: 97 active+clean; 144 MiB data, 394 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 982 op/s 2024-04-07T14:08:56.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 10751 21.40 MB/sec warmup 18 sec latency 46.450 ms 2024-04-07T14:08:56.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:56 smithi145 ceph-mon[26800]: pgmap v246: 97 pgs: 97 active+clean; 144 MiB data, 394 MiB used, 715 GiB / 715 GiB avail; 7.9 KiB/s rd, 13 MiB/s wr, 982 op/s 2024-04-07T14:08:57.420 INFO:teuthology.orchestra.run.smithi111.stderr:module 'nfs' is already enabled 2024-04-07T14:08:57.491 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:57 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2310437194' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-07T14:08:57.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 11250 20.64 MB/sec warmup 19 sec latency 26.286 ms 2024-04-07T14:08:57.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:57 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2310437194' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "nfs", "force": true}]: dispatch 2024-04-07T14:08:58.495 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:58 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2310437194' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-07T14:08:58.495 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:58 smithi111 ceph-mon[24802]: mgrmap e21: smithi111.rtrlso(active, since 6m), standbys: smithi145.qiunpb 2024-04-07T14:08:58.495 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:08:58 smithi111 ceph-mon[24802]: pgmap v247: 97 pgs: 97 active+clean; 144 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 901 op/s 2024-04-07T14:08:58.518 INFO:teuthology.orchestra.run.smithi111.stdout: 5 11814 20.12 MB/sec warmup 20 sec latency 24.008 ms 2024-04-07T14:08:58.650 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1' 2024-04-07T14:08:58.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:58 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2310437194' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "nfs", "force": true}]': finished 2024-04-07T14:08:58.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:58 smithi145 ceph-mon[26800]: mgrmap e21: smithi111.rtrlso(active, since 6m), standbys: smithi145.qiunpb 2024-04-07T14:08:58.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:08:58 smithi145 ceph-mon[26800]: pgmap v247: 97 pgs: 97 active+clean; 144 MiB data, 404 MiB used, 715 GiB / 715 GiB avail; 13 KiB/s rd, 9.7 MiB/s wr, 901 op/s 2024-04-07T14:08:59.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 12519 19.93 MB/sec warmup 21 sec latency 56.299 ms 2024-04-07T14:09:00.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 13089 19.45 MB/sec warmup 22 sec latency 34.412 ms 2024-04-07T14:09:00.778 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:00 smithi111 ceph-mon[24802]: pgmap v248: 97 pgs: 97 active+clean; 156 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.06k op/s 2024-04-07T14:09:01.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:00 smithi145 ceph-mon[26800]: pgmap v248: 97 pgs: 97 active+clean; 156 MiB data, 437 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.06k op/s 2024-04-07T14:09:01.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 13682 19.74 MB/sec warmup 23 sec latency 33.473 ms 2024-04-07T14:09:02.044 INFO:teuthology.orchestra.run.smithi111.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T14:09:02.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 14173 19.76 MB/sec warmup 24 sec latency 29.426 ms 2024-04-07T14:09:02.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:02 smithi145 ceph-mon[26800]: pgmap v249: 97 pgs: 97 active+clean; 156 MiB data, 445 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 753 op/s 2024-04-07T14:09:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:09:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:09:02.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:02 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:09:02.797 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:02 smithi111 ceph-mon[24802]: pgmap v249: 97 pgs: 97 active+clean; 156 MiB data, 445 MiB used, 715 GiB / 715 GiB avail; 12 KiB/s rd, 8.3 MiB/s wr, 753 op/s 2024-04-07T14:09:02.797 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:02.797 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:09:02.797 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:09:02.797 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:02 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:09:03.129 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-04-07T14:09:03.130 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:09:03.130 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done' 2024-04-07T14:09:03.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 14636 19.46 MB/sec warmup 25 sec latency 29.983 ms 2024-04-07T14:09:03.666 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:03 smithi111 ceph-mon[24802]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:03.666 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:03 smithi111 ceph-mon[24802]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T14:09:03.666 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:03 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:03.666 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:03 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:03 smithi145 ceph-mon[26800]: from='client.14602 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:03 smithi145 ceph-mon[26800]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T14:09:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:03 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:04.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:03 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:04.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 15310 19.14 MB/sec warmup 26 sec latency 19.457 ms 2024-04-07T14:09:04.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:04 smithi111 ceph-mon[24802]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T14:09:04.806 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:04 smithi111 ceph-mon[24802]: pgmap v250: 97 pgs: 97 active+clean; 160 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 889 op/s 2024-04-07T14:09:05.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:04 smithi145 ceph-mon[26800]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d 2024-04-07T14:09:05.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:04 smithi145 ceph-mon[26800]: pgmap v250: 97 pgs: 97 active+clean; 160 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 17 KiB/s rd, 9.6 MiB/s wr, 889 op/s 2024-04-07T14:09:05.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 16217 19.11 MB/sec warmup 27 sec latency 32.832 ms 2024-04-07T14:09:06.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 16971 19.24 MB/sec warmup 28 sec latency 18.499 ms 2024-04-07T14:09:06.566 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:06 smithi111 ceph-mon[24802]: pgmap v251: 97 pgs: 97 active+clean; 167 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 897 op/s 2024-04-07T14:09:06.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:06 smithi145 ceph-mon[26800]: pgmap v251: 97 pgs: 97 active+clean; 167 MiB data, 521 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 9.6 MiB/s wr, 897 op/s 2024-04-07T14:09:07.477 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-04-07T14:09:07.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 17622 19.50 MB/sec warmup 29 sec latency 34.453 ms 2024-04-07T14:09:08.486 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (5m) 32s ago 6m 21.5M - ba2b418f427c 012d9a12b505 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (6m) 32s ago 6m 6400k - 17.2.0 e1d6a67b021e 0eff85d2a49e 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi145 smithi145 running (6m) 34s ago 6m 6405k - 17.2.0 e1d6a67b021e 9e913164ca86 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (5m) 32s ago 6m 53.3M - 8.3.5 dad864ee21e9 7d5afae8bf58 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi111.pvxsdw smithi111 running (64s) 32s ago 63s 15.8M - 17.2.0 e1d6a67b021e 75df9269a572 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi145.fczrwb smithi145 running (61s) 34s ago 61s 14.3M - 17.2.0 e1d6a67b021e c46e26578d78 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.rtrlso smithi111 *:9283 running (8m) 32s ago 8m 460M - 17.2.0 e1d6a67b021e 94ef2bc4f582 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi145.qiunpb smithi145 *:8443,9283 running (6m) 34s ago 6m 413M - 17.2.0 e1d6a67b021e 7da4bd63d481 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (8m) 32s ago 8m 49.2M 2048M 17.2.0 e1d6a67b021e 382c73ff91e2 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi145 smithi145 running (6m) 34s ago 6m 34.1M 2048M 17.2.0 e1d6a67b021e c8daa5dcac8e 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.0.0.smithi111.jegwqs smithi111 *:2049 running (42s) 32s ago 42s 54.1M - 4.0 e1d6a67b021e 31e257496f13 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.1.0.smithi145.lsscmc smithi145 *:2049 running (37s) 34s ago 37s 46.5M - 4.0 e1d6a67b021e de97e4d27131 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (6m) 32s ago 6m 15.8M - 1dbe0e931976 ba6021f0730f 2024-04-07T14:09:08.487 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi145 smithi145 *:9100 running (6m) 34s ago 6m 18.7M - 1dbe0e931976 bf68fb6f0950 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (5m) 32s ago 5m 42.2M 3216M 17.2.0 e1d6a67b021e ea5afbacec5e 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 32s ago 5m 45.5M 3216M 17.2.0 e1d6a67b021e 1da0bd36d060 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (4m) 32s ago 4m 39.3M 3216M 17.2.0 e1d6a67b021e c6ab9df37c92 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi111 running (4m) 32s ago 4m 40.7M 3216M 17.2.0 e1d6a67b021e 195f5ab7aba6 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi145 running (3m) 34s ago 3m 39.3M 3984M 17.2.0 e1d6a67b021e 06ffbb78268c 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi145 running (3m) 34s ago 3m 40.2M 3984M 17.2.0 e1d6a67b021e 5ffb10ce0799 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 smithi145 running (2m) 34s ago 2m 39.7M 3984M 17.2.0 e1d6a67b021e 5e283a14fbad 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 smithi145 running (2m) 34s ago 2m 43.7M 3984M 17.2.0 e1d6a67b021e d5503daa3f47 2024-04-07T14:09:08.488 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (5m) 32s ago 5m 62.5M - 514e6a882f6e 1965f3714193 2024-04-07T14:09:08.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 18074 19.39 MB/sec warmup 30 sec latency 28.098 ms 2024-04-07T14:09:08.838 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:08 smithi111 ceph-mon[24802]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:08.839 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:08 smithi111 ceph-mon[24802]: pgmap v252: 97 pgs: 97 active+clean; 167 MiB data, 527 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 7.7 MiB/s wr, 683 op/s 2024-04-07T14:09:09.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:08 smithi145 ceph-mon[26800]: from='client.14606 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:08 smithi145 ceph-mon[26800]: pgmap v252: 97 pgs: 97 active+clean; 167 MiB data, 527 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 7.7 MiB/s wr, 683 op/s 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:09.089 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-07T14:09:09.090 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:09:09.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 18556 18.90 MB/sec warmup 31 sec latency 23.370 ms 2024-04-07T14:09:09.628 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:09:09.628 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d", 2024-04-07T14:09:09.628 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-04-07T14:09:09.628 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-04-07T14:09:09.628 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "", 2024-04-07T14:09:09.629 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:e5c885fe95ee982ff7b45214daa61b1a6d09d57d image" 2024-04-07T14:09:09.629 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:09:09.842 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:09 smithi111 ceph-mon[24802]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:09.843 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:09 smithi111 ceph-mon[24802]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:09.843 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:09 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/2439776635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:09:10.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:09 smithi145 ceph-mon[26800]: from='client.24347 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:10.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:09 smithi145 ceph-mon[26800]: from='client.24351 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:10.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:09 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/2439776635' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:09:10.272 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-04-07T14:09:10.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 19259 18.92 MB/sec warmup 32 sec latency 22.526 ms 2024-04-07T14:09:10.740 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:10 smithi111 ceph-mon[24802]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 949 op/s 2024-04-07T14:09:10.740 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:10 smithi111 ceph-mon[24802]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:10.740 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:10 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/4250286184' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:09:11.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:10 smithi145 ceph-mon[26800]: pgmap v253: 97 pgs: 97 active+clean; 174 MiB data, 581 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 10 MiB/s wr, 949 op/s 2024-04-07T14:09:11.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:10 smithi145 ceph-mon[26800]: from='client.24357 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:11.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:10 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/4250286184' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:09:11.519 INFO:teuthology.orchestra.run.smithi111.stdout: 5 20195 18.80 MB/sec warmup 33 sec latency 14.960 ms 2024-04-07T14:09:12.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 20907 19.16 MB/sec warmup 34 sec latency 23.965 ms 2024-04-07T14:09:12.761 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:12 smithi111 ceph-mon[24802]: pgmap v254: 97 pgs: 97 active+clean; 179 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 787 op/s 2024-04-07T14:09:13.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:12 smithi145 ceph-mon[26800]: pgmap v254: 97 pgs: 97 active+clean; 179 MiB data, 591 MiB used, 715 GiB / 715 GiB avail; 10 KiB/s rd, 8.2 MiB/s wr, 787 op/s 2024-04-07T14:09:13.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 21560 19.29 MB/sec warmup 35 sec latency 33.294 ms 2024-04-07T14:09:14.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 22027 18.89 MB/sec warmup 36 sec latency 20.918 ms 2024-04-07T14:09:14.770 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:14 smithi111 ceph-mon[24802]: pgmap v255: 97 pgs: 97 active+clean; 185 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-04-07T14:09:15.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:14 smithi145 ceph-mon[26800]: pgmap v255: 97 pgs: 97 active+clean; 185 MiB data, 655 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 10 MiB/s wr, 1.01k op/s 2024-04-07T14:09:15.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 22802 18.88 MB/sec warmup 37 sec latency 18.271 ms 2024-04-07T14:09:16.521 INFO:teuthology.orchestra.run.smithi111.stdout: 5 23767 18.87 MB/sec warmup 38 sec latency 18.972 ms 2024-04-07T14:09:16.527 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:16 smithi111 ceph-mon[24802]: pgmap v256: 97 pgs: 97 active+clean; 185 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 929 op/s 2024-04-07T14:09:16.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:16 smithi145 ceph-mon[26800]: pgmap v256: 97 pgs: 97 active+clean; 185 MiB data, 691 MiB used, 715 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 929 op/s 2024-04-07T14:09:17.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 24694 19.34 MB/sec warmup 39 sec latency 27.846 ms 2024-04-07T14:09:18.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 25316 19.34 MB/sec warmup 40 sec latency 21.778 ms 2024-04-07T14:09:18.787 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:18 smithi111 ceph-mon[24802]: pgmap v257: 97 pgs: 97 active+clean; 188 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 800 op/s 2024-04-07T14:09:19.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:18 smithi145 ceph-mon[26800]: pgmap v257: 97 pgs: 97 active+clean; 188 MiB data, 716 MiB used, 715 GiB / 715 GiB avail; 16 KiB/s rd, 8.8 MiB/s wr, 800 op/s 2024-04-07T14:09:19.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 26104 19.25 MB/sec warmup 41 sec latency 19.889 ms 2024-04-07T14:09:20.520 INFO:teuthology.orchestra.run.smithi111.stdout: 5 26864 19.12 MB/sec warmup 42 sec latency 31.270 ms 2024-04-07T14:09:20.800 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:20 smithi111 ceph-mon[24802]: pgmap v258: 97 pgs: 97 active+clean; 200 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-07T14:09:21.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:20 smithi145 ceph-mon[26800]: pgmap v258: 97 pgs: 97 active+clean; 200 MiB data, 795 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.10k op/s 2024-04-07T14:09:21.528 INFO:teuthology.orchestra.run.smithi111.stdout: 5 27504 19.06 MB/sec warmup 43 sec latency 38.725 ms 2024-04-07T14:09:22.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 28235 19.30 MB/sec warmup 44 sec latency 22.526 ms 2024-04-07T14:09:22.809 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:22 smithi111 ceph-mon[24802]: pgmap v259: 97 pgs: 97 active+clean; 200 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 833 op/s 2024-04-07T14:09:23.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:22 smithi145 ceph-mon[26800]: pgmap v259: 97 pgs: 97 active+clean; 200 MiB data, 784 MiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.1 MiB/s wr, 833 op/s 2024-04-07T14:09:23.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 28825 19.31 MB/sec warmup 45 sec latency 32.624 ms 2024-04-07T14:09:24.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 29448 19.09 MB/sec warmup 46 sec latency 26.638 ms 2024-04-07T14:09:24.831 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:24 smithi111 ceph-mon[24802]: pgmap v260: 97 pgs: 97 active+clean; 206 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 934 op/s 2024-04-07T14:09:25.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:24 smithi145 ceph-mon[26800]: pgmap v260: 97 pgs: 97 active+clean; 206 MiB data, 838 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 934 op/s 2024-04-07T14:09:25.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 30181 19.09 MB/sec warmup 47 sec latency 29.106 ms 2024-04-07T14:09:26.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 30854 18.90 MB/sec warmup 48 sec latency 29.462 ms 2024-04-07T14:09:26.622 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:26 smithi111 ceph-mon[24802]: pgmap v261: 97 pgs: 97 active+clean; 210 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-07T14:09:26.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:26 smithi145 ceph-mon[26800]: pgmap v261: 97 pgs: 97 active+clean; 210 MiB data, 880 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 882 op/s 2024-04-07T14:09:27.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 31683 19.24 MB/sec warmup 49 sec latency 22.158 ms 2024-04-07T14:09:28.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 32329 19.28 MB/sec warmup 50 sec latency 28.610 ms 2024-04-07T14:09:28.673 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:28 smithi111 ceph-mon[24802]: pgmap v262: 97 pgs: 97 active+clean; 211 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 838 op/s 2024-04-07T14:09:29.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:28 smithi145 ceph-mon[26800]: pgmap v262: 97 pgs: 97 active+clean; 211 MiB data, 883 MiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 838 op/s 2024-04-07T14:09:29.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 32904 19.03 MB/sec warmup 51 sec latency 26.943 ms 2024-04-07T14:09:30.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 33694 19.07 MB/sec warmup 52 sec latency 28.888 ms 2024-04-07T14:09:30.683 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:30 smithi111 ceph-mon[24802]: pgmap v263: 97 pgs: 97 active+clean; 217 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-07T14:09:31.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:30 smithi145 ceph-mon[26800]: pgmap v263: 97 pgs: 97 active+clean; 217 MiB data, 939 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.05k op/s 2024-04-07T14:09:31.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 34607 19.08 MB/sec warmup 53 sec latency 18.333 ms 2024-04-07T14:09:32.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 35510 19.42 MB/sec warmup 54 sec latency 18.537 ms 2024-04-07T14:09:32.693 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:32 smithi111 ceph-mon[24802]: pgmap v264: 97 pgs: 97 active+clean; 220 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-04-07T14:09:33.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:32 smithi145 ceph-mon[26800]: pgmap v264: 97 pgs: 97 active+clean; 220 MiB data, 940 MiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 8.5 MiB/s wr, 747 op/s 2024-04-07T14:09:33.532 INFO:teuthology.orchestra.run.smithi111.stdout: 5 36128 19.31 MB/sec warmup 55 sec latency 21.272 ms 2024-04-07T14:09:34.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 36808 19.20 MB/sec warmup 56 sec latency 27.603 ms 2024-04-07T14:09:34.707 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:34 smithi111 ceph-mon[24802]: pgmap v265: 97 pgs: 97 active+clean; 231 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 972 op/s 2024-04-07T14:09:35.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:34 smithi145 ceph-mon[26800]: pgmap v265: 97 pgs: 97 active+clean; 231 MiB data, 1018 MiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 11 MiB/s wr, 972 op/s 2024-04-07T14:09:35.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 37492 19.13 MB/sec warmup 57 sec latency 33.632 ms 2024-04-07T14:09:36.467 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:36 smithi111 ceph-mon[24802]: pgmap v266: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 966 op/s 2024-04-07T14:09:36.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 38213 19.12 MB/sec warmup 58 sec latency 45.955 ms 2024-04-07T14:09:36.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:36 smithi145 ceph-mon[26800]: pgmap v266: 97 pgs: 97 active+clean; 232 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 966 op/s 2024-04-07T14:09:37.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 39105 19.41 MB/sec warmup 59 sec latency 38.002 ms 2024-04-07T14:09:37.722 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:09:37.722 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:37 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:09:38.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/trash_purge_schedule"}]: dispatch 2024-04-07T14:09:38.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:37 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi111.rtrlso/mirror_snapshot_schedule"}]: dispatch 2024-04-07T14:09:38.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 39738 19.32 MB/sec warmup 60 sec latency 28.798 ms 2024-04-07T14:09:38.730 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:38 smithi111 ceph-mon[24802]: pgmap v267: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 848 op/s 2024-04-07T14:09:39.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:38 smithi145 ceph-mon[26800]: pgmap v267: 97 pgs: 97 active+clean; 233 MiB data, 1.0 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 9.4 MiB/s wr, 848 op/s 2024-04-07T14:09:39.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 40451 19.23 MB/sec warmup 61 sec latency 32.142 ms 2024-04-07T14:09:40.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 41165 19.15 MB/sec warmup 62 sec latency 27.037 ms 2024-04-07T14:09:40.745 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:40 smithi111 ceph-mon[24802]: pgmap v268: 97 pgs: 97 active+clean; 241 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-07T14:09:40.745 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:40.746 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:09:40.746 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:40 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:40.809 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:40 smithi145 ceph-mon[26800]: pgmap v268: 97 pgs: 97 active+clean; 241 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 12 MiB/s wr, 1.09k op/s 2024-04-07T14:09:40.809 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:40 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:40.809 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:40 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:09:40.809 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:40 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:09:40.830 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-04-07T14:09:41.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 41752 19.11 MB/sec warmup 63 sec latency 22.623 ms 2024-04-07T14:09:41.750 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:41 smithi111 ceph-mon[24802]: Upgrade: Target is version 19.0.0-2578-ge5c885fe (unknown) 2024-04-07T14:09:41.750 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:41 smithi111 ceph-mon[24802]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1'] 2024-04-07T14:09:41.751 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:41 smithi111 ceph-mon[24802]: Upgrade: Need to upgrade myself (mgr.smithi111.rtrlso) 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 65s ago 7m 21.5M - ba2b418f427c 012d9a12b505 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (7m) 65s ago 7m 6400k - 17.2.0 e1d6a67b021e 0eff85d2a49e 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi145 smithi145 running (6m) 67s ago 6m 6405k - 17.2.0 e1d6a67b021e 9e913164ca86 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (5m) 65s ago 7m 53.3M - 8.3.5 dad864ee21e9 7d5afae8bf58 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi111.pvxsdw smithi111 running (97s) 65s ago 97s 15.8M - 17.2.0 e1d6a67b021e 75df9269a572 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi145.fczrwb smithi145 running (95s) 67s ago 94s 14.3M - 17.2.0 e1d6a67b021e c46e26578d78 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.rtrlso smithi111 *:9283 running (8m) 65s ago 8m 460M - 17.2.0 e1d6a67b021e 94ef2bc4f582 2024-04-07T14:09:41.910 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi145.qiunpb smithi145 *:8443,9283 running (6m) 67s ago 6m 413M - 17.2.0 e1d6a67b021e 7da4bd63d481 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (8m) 65s ago 8m 49.2M 2048M 17.2.0 e1d6a67b021e 382c73ff91e2 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi145 smithi145 running (6m) 67s ago 6m 34.1M 2048M 17.2.0 e1d6a67b021e c8daa5dcac8e 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.0.0.smithi111.jegwqs smithi111 *:2049 running (76s) 65s ago 75s 54.1M - 4.0 e1d6a67b021e 31e257496f13 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.1.0.smithi145.lsscmc smithi145 *:2049 running (70s) 67s ago 70s 46.5M - 4.0 e1d6a67b021e de97e4d27131 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 65s ago 7m 15.8M - 1dbe0e931976 ba6021f0730f 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi145 smithi145 *:9100 running (6m) 67s ago 6m 18.7M - 1dbe0e931976 bf68fb6f0950 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 65s ago 6m 42.2M 3216M 17.2.0 e1d6a67b021e ea5afbacec5e 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (5m) 65s ago 5m 45.5M 3216M 17.2.0 e1d6a67b021e 1da0bd36d060 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 65s ago 5m 39.3M 3216M 17.2.0 e1d6a67b021e c6ab9df37c92 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi111 running (4m) 65s ago 4m 40.7M 3216M 17.2.0 e1d6a67b021e 195f5ab7aba6 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi145 running (4m) 67s ago 4m 39.3M 3984M 17.2.0 e1d6a67b021e 06ffbb78268c 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi145 running (3m) 67s ago 3m 40.2M 3984M 17.2.0 e1d6a67b021e 5ffb10ce0799 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 smithi145 running (3m) 67s ago 3m 39.7M 3984M 17.2.0 e1d6a67b021e 5e283a14fbad 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 smithi145 running (3m) 67s ago 3m 43.7M 3984M 17.2.0 e1d6a67b021e d5503daa3f47 2024-04-07T14:09:41.911 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (6m) 65s ago 6m 62.5M - 514e6a882f6e 1965f3714193 2024-04-07T14:09:42.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:41 smithi145 ceph-mon[26800]: Upgrade: Target is version 19.0.0-2578-ge5c885fe (unknown) 2024-04-07T14:09:42.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:41 smithi145 ceph-mon[26800]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1, digests ['quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1'] 2024-04-07T14:09:42.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:41 smithi145 ceph-mon[26800]: Upgrade: Need to upgrade myself (mgr.smithi111.rtrlso) 2024-04-07T14:09:42.469 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:09:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-04-07T14:09:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:42.469 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-07T14:09:42.470 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:09:42.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 42381 19.24 MB/sec warmup 64 sec latency 30.081 ms 2024-04-07T14:09:42.765 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:42 smithi111 ceph-mon[24802]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:42.765 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:42 smithi111 ceph-mon[24802]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1 on smithi145 2024-04-07T14:09:42.765 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:42 smithi111 ceph-mon[24802]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:42.765 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:42 smithi111 ceph-mon[24802]: pgmap v269: 97 pgs: 97 active+clean; 245 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-04-07T14:09:42.765 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:42 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/236605437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:09:42.938 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:09:42.938 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1", 2024-04-07T14:09:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-04-07T14:09:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-04-07T14:09:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "0/23 daemons upgraded", 2024-04-07T14:09:42.939 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1 image on host smithi145" 2024-04-07T14:09:42.939 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:09:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:42 smithi145 ceph-mon[26800]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:42 smithi145 ceph-mon[26800]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1 on smithi145 2024-04-07T14:09:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:42 smithi145 ceph-mon[26800]: from='client.14630 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:42 smithi145 ceph-mon[26800]: pgmap v269: 97 pgs: 97 active+clean; 245 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 9.4 MiB/s wr, 861 op/s 2024-04-07T14:09:43.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:42 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/236605437' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:09:43.492 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-04-07T14:09:43.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 42915 19.22 MB/sec warmup 65 sec latency 32.263 ms 2024-04-07T14:09:43.770 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:43 smithi111 ceph-mon[24802]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:43.770 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:43 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/88180900' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:09:44.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:43 smithi145 ceph-mon[26800]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:44.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:43 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/88180900' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:09:44.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 43362 19.03 MB/sec warmup 66 sec latency 26.809 ms 2024-04-07T14:09:44.689 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:44 smithi111 ceph-mon[24802]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:44.689 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:44 smithi111 ceph-mon[24802]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-07T14:09:45.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:44 smithi145 ceph-mon[26800]: from='client.14642 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:09:45.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:44 smithi145 ceph-mon[26800]: pgmap v270: 97 pgs: 97 active+clean; 249 MiB data, 1.1 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 11 MiB/s wr, 1.07k op/s 2024-04-07T14:09:45.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 44169 18.99 MB/sec warmup 67 sec latency 29.984 ms 2024-04-07T14:09:46.435 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:46 smithi111 ceph-mon[24802]: pgmap v271: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 967 op/s 2024-04-07T14:09:46.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 44965 18.95 MB/sec warmup 68 sec latency 28.387 ms 2024-04-07T14:09:46.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:46 smithi145 ceph-mon[26800]: pgmap v271: 97 pgs: 97 active+clean; 255 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 967 op/s 2024-04-07T14:09:47.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 45721 19.13 MB/sec warmup 69 sec latency 28.500 ms 2024-04-07T14:09:48.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 46502 19.22 MB/sec warmup 70 sec latency 26.239 ms 2024-04-07T14:09:48.701 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:48 smithi111 ceph-mon[24802]: pgmap v272: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 8.8 MiB/s wr, 823 op/s 2024-04-07T14:09:49.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:48 smithi145 ceph-mon[26800]: pgmap v272: 97 pgs: 97 active+clean; 254 MiB data, 1.2 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 8.8 MiB/s wr, 823 op/s 2024-04-07T14:09:49.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 47331 19.16 MB/sec warmup 71 sec latency 20.840 ms 2024-04-07T14:09:50.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 48276 19.14 MB/sec warmup 72 sec latency 22.416 ms 2024-04-07T14:09:50.710 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:50 smithi111 ceph-mon[24802]: pgmap v273: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-04-07T14:09:51.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:50 smithi145 ceph-mon[26800]: pgmap v273: 97 pgs: 97 active+clean; 270 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 12 MiB/s wr, 1.13k op/s 2024-04-07T14:09:51.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 49264 19.41 MB/sec warmup 73 sec latency 26.133 ms 2024-04-07T14:09:52.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 50100 19.49 MB/sec warmup 74 sec latency 49.194 ms 2024-04-07T14:09:52.721 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:52 smithi111 ceph-mon[24802]: pgmap v274: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 873 op/s 2024-04-07T14:09:53.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:52 smithi145 ceph-mon[26800]: pgmap v274: 97 pgs: 97 active+clean; 267 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.9 MiB/s wr, 873 op/s 2024-04-07T14:09:53.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 50734 19.36 MB/sec warmup 75 sec latency 91.178 ms 2024-04-07T14:09:54.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 51524 19.37 MB/sec warmup 76 sec latency 28.732 ms 2024-04-07T14:09:54.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:54 smithi111 ceph-mon[24802]: pgmap v275: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 948 op/s 2024-04-07T14:09:55.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:54 smithi145 ceph-mon[26800]: pgmap v275: 97 pgs: 97 active+clean; 269 MiB data, 1.3 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 948 op/s 2024-04-07T14:09:55.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 52437 19.37 MB/sec warmup 77 sec latency 28.071 ms 2024-04-07T14:09:56.507 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:56 smithi111 ceph-mon[24802]: pgmap v276: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-07T14:09:56.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 53336 19.62 MB/sec warmup 78 sec latency 21.332 ms 2024-04-07T14:09:56.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:56 smithi145 ceph-mon[26800]: pgmap v276: 97 pgs: 97 active+clean; 280 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 985 op/s 2024-04-07T14:09:57.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 53922 19.52 MB/sec warmup 79 sec latency 22.257 ms 2024-04-07T14:09:58.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 54669 19.44 MB/sec warmup 80 sec latency 28.204 ms 2024-04-07T14:09:58.778 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:09:58 smithi111 ceph-mon[24802]: pgmap v277: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 966 op/s 2024-04-07T14:09:59.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:09:58 smithi145 ceph-mon[26800]: pgmap v277: 97 pgs: 97 active+clean; 284 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 10 MiB/s wr, 966 op/s 2024-04-07T14:09:59.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 55369 19.39 MB/sec warmup 81 sec latency 34.628 ms 2024-04-07T14:10:00.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 56081 19.39 MB/sec warmup 82 sec latency 21.087 ms 2024-04-07T14:10:00.862 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:00 smithi111 ceph-mon[24802]: pgmap v278: 97 pgs: 97 active+clean; 290 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-04-07T14:10:00.862 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:00 smithi111 ceph-mon[24802]: overall HEALTH_OK 2024-04-07T14:10:01.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:00 smithi145 ceph-mon[26800]: pgmap v278: 97 pgs: 97 active+clean; 290 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 13 MiB/s wr, 1.22k op/s 2024-04-07T14:10:01.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:00 smithi145 ceph-mon[26800]: overall HEALTH_OK 2024-04-07T14:10:01.533 INFO:teuthology.orchestra.run.smithi111.stdout: 5 56800 19.52 MB/sec warmup 83 sec latency 22.735 ms 2024-04-07T14:10:02.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 57342 19.46 MB/sec warmup 84 sec latency 28.092 ms 2024-04-07T14:10:02.882 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:02 smithi111 ceph-mon[24802]: pgmap v279: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 957 op/s 2024-04-07T14:10:03.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:02 smithi145 ceph-mon[26800]: pgmap v279: 97 pgs: 97 active+clean; 293 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 10 MiB/s wr, 957 op/s 2024-04-07T14:10:03.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 57828 19.36 MB/sec warmup 85 sec latency 37.292 ms 2024-04-07T14:10:04.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 58665 19.35 MB/sec warmup 86 sec latency 32.645 ms 2024-04-07T14:10:04.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:04 smithi145 ceph-mon[26800]: pgmap v280: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-04-07T14:10:04.976 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:04 smithi111 ceph-mon[24802]: pgmap v280: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 12 MiB/s wr, 1.14k op/s 2024-04-07T14:10:05.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 59401 19.31 MB/sec warmup 87 sec latency 255.600 ms 2024-04-07T14:10:06.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 60058 19.42 MB/sec warmup 88 sec latency 30.675 ms 2024-04-07T14:10:06.653 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:06 smithi111 ceph-mon[24802]: pgmap v281: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-07T14:10:06.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:06 smithi145 ceph-mon[26800]: pgmap v281: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 KiB/s rd, 11 MiB/s wr, 1.05k op/s 2024-04-07T14:10:07.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 60758 19.43 MB/sec warmup 89 sec latency 18.957 ms 2024-04-07T14:10:08.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 61405 19.35 MB/sec warmup 90 sec latency 20.019 ms 2024-04-07T14:10:08.666 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:08 smithi111 ceph-mon[24802]: pgmap v282: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 8.8 MiB/s wr, 807 op/s 2024-04-07T14:10:09.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:08 smithi145 ceph-mon[26800]: pgmap v282: 97 pgs: 97 active+clean; 300 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 KiB/s rd, 8.8 MiB/s wr, 807 op/s 2024-04-07T14:10:09.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 62235 19.34 MB/sec warmup 91 sec latency 18.225 ms 2024-04-07T14:10:10.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 63021 19.32 MB/sec warmup 92 sec latency 24.510 ms 2024-04-07T14:10:10.726 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:10 smithi111 ceph-mon[24802]: pgmap v283: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-07T14:10:10.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:10 smithi145 ceph-mon[26800]: pgmap v283: 97 pgs: 97 active+clean; 311 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 11 MiB/s wr, 1.01k op/s 2024-04-07T14:10:11.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 63395 19.27 MB/sec warmup 93 sec latency 235.744 ms 2024-04-07T14:10:12.291 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: Upgrade: Updating mgr.smithi145.qiunpb 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: Deploying daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:10:12.292 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:12 smithi111 ceph-mon[24802]: pgmap v284: 97 pgs: 97 active+clean; 316 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 826 op/s 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: Upgrade: Updating mgr.smithi145.qiunpb 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi145.qiunpb", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: Deploying daemon mgr.smithi145.qiunpb on smithi145 2024-04-07T14:10:12.525 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:12 smithi145 ceph-mon[26800]: pgmap v284: 97 pgs: 97 active+clean; 316 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 8.7 MiB/s wr, 826 op/s 2024-04-07T14:10:12.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 63741 19.23 MB/sec warmup 94 sec latency 45.468 ms 2024-04-07T14:10:13.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 64274 19.19 MB/sec warmup 95 sec latency 109.283 ms 2024-04-07T14:10:14.022 INFO:teuthology.orchestra.run.smithi111.stdout:true 2024-04-07T14:10:14.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 64653 19.06 MB/sec warmup 96 sec latency 41.600 ms 2024-04-07T14:10:14.817 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:14 smithi111 ceph-mon[24802]: pgmap v285: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 939 op/s 2024-04-07T14:10:15.001 INFO:teuthology.orchestra.run.smithi111.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:alertmanager.smithi111 smithi111 *:9093,9094 running (6m) 98s ago 8m 21.5M - ba2b418f427c 012d9a12b505 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi111 smithi111 running (8m) 98s ago 8m 6400k - 17.2.0 e1d6a67b021e 0eff85d2a49e 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:crash.smithi145 smithi145 running (7m) 100s ago 7m 6405k - 17.2.0 e1d6a67b021e 9e913164ca86 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:grafana.smithi111 smithi111 *:3000 running (6m) 98s ago 7m 53.3M - 8.3.5 dad864ee21e9 7d5afae8bf58 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi111.pvxsdw smithi111 running (2m) 98s ago 2m 15.8M - 17.2.0 e1d6a67b021e 75df9269a572 2024-04-07T14:10:15.002 INFO:teuthology.orchestra.run.smithi111.stdout:mds.foofs.smithi145.fczrwb smithi145 running (2m) 100s ago 2m 14.3M - 17.2.0 e1d6a67b021e c46e26578d78 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi111.rtrlso smithi111 *:9283 running (9m) 98s ago 9m 460M - 17.2.0 e1d6a67b021e 94ef2bc4f582 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:mgr.smithi145.qiunpb smithi145 *:8443,9283 running (7m) 100s ago 7m 413M - 17.2.0 e1d6a67b021e 7da4bd63d481 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi111 smithi111 running (9m) 98s ago 9m 49.2M 2048M 17.2.0 e1d6a67b021e 382c73ff91e2 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:mon.smithi145 smithi145 running (7m) 100s ago 7m 34.1M 2048M 17.2.0 e1d6a67b021e c8daa5dcac8e 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.0.0.smithi111.jegwqs smithi111 *:2049 running (109s) 98s ago 108s 54.1M - 4.0 e1d6a67b021e 31e257496f13 2024-04-07T14:10:15.003 INFO:teuthology.orchestra.run.smithi111.stdout:nfs.foo.1.0.smithi145.lsscmc smithi145 *:2049 running (104s) 100s ago 103s 46.5M - 4.0 e1d6a67b021e de97e4d27131 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi111 smithi111 *:9100 running (7m) 98s ago 7m 15.8M - 1dbe0e931976 ba6021f0730f 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:node-exporter.smithi145 smithi145 *:9100 running (7m) 100s ago 7m 18.7M - 1dbe0e931976 bf68fb6f0950 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:osd.0 smithi111 running (6m) 98s ago 6m 42.2M 3216M 17.2.0 e1d6a67b021e ea5afbacec5e 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:osd.1 smithi111 running (6m) 98s ago 6m 45.5M 3216M 17.2.0 e1d6a67b021e 1da0bd36d060 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:osd.2 smithi111 running (5m) 98s ago 5m 39.3M 3216M 17.2.0 e1d6a67b021e c6ab9df37c92 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:osd.3 smithi111 running (5m) 98s ago 5m 40.7M 3216M 17.2.0 e1d6a67b021e 195f5ab7aba6 2024-04-07T14:10:15.004 INFO:teuthology.orchestra.run.smithi111.stdout:osd.4 smithi145 running (4m) 100s ago 4m 39.3M 3984M 17.2.0 e1d6a67b021e 06ffbb78268c 2024-04-07T14:10:15.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.5 smithi145 running (4m) 100s ago 4m 40.2M 3984M 17.2.0 e1d6a67b021e 5ffb10ce0799 2024-04-07T14:10:15.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.6 smithi145 running (4m) 100s ago 4m 39.7M 3984M 17.2.0 e1d6a67b021e 5e283a14fbad 2024-04-07T14:10:15.005 INFO:teuthology.orchestra.run.smithi111.stdout:osd.7 smithi145 running (3m) 100s ago 3m 43.7M 3984M 17.2.0 e1d6a67b021e d5503daa3f47 2024-04-07T14:10:15.005 INFO:teuthology.orchestra.run.smithi111.stdout:prometheus.smithi111 smithi111 *:9095 running (6m) 98s ago 6m 62.5M - 514e6a882f6e 1965f3714193 2024-04-07T14:10:15.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:14 smithi145 ceph-mon[26800]: pgmap v285: 97 pgs: 97 active+clean; 323 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 17 KiB/s rd, 10 MiB/s wr, 939 op/s 2024-04-07T14:10:15.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 65183 18.96 MB/sec warmup 97 sec latency 27.190 ms 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mon": { 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "mgr": { 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:10:15.635 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "osd": { 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 8 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "mds": { 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 2 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: }, 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "overall": { 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: "ceph version 17.2.0 (43e2e60a7559d3f46c9d53f1ca875fd499a1e35e) quincy (stable)": 14 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout: } 2024-04-07T14:10:15.636 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:10:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:15 smithi145 ceph-mon[26800]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:15 smithi145 ceph-mon[26800]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:15.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:15 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/1793401188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:10:15.821 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:15 smithi111 ceph-mon[24802]: from='client.24389 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:15.821 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:15 smithi111 ceph-mon[24802]: from='client.24393 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:15.821 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:15 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/1793401188' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout:{ 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:72637ed65c3eacb208a01eb8fcf572a65857060627154597bacdb87f092e5fe1", 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout: "in_progress": true, 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout: "services_complete": [], 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout: "progress": "0/23 daemons upgraded", 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout: "message": "Currently upgrading mgr daemons" 2024-04-07T14:10:16.076 INFO:teuthology.orchestra.run.smithi111.stdout:} 2024-04-07T14:10:16.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 65787 18.94 MB/sec warmup 98 sec latency 48.274 ms 2024-04-07T14:10:16.574 INFO:teuthology.orchestra.run.smithi111.stdout:HEALTH_OK 2024-04-07T14:10:16.827 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:16 smithi111 ceph-mon[24802]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:16.827 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:16 smithi111 ceph-mon[24802]: pgmap v286: 97 pgs: 97 active+clean; 322 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 798 op/s 2024-04-07T14:10:16.828 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:16 smithi111 ceph-mon[24802]: from='client.? 172.21.15.111:0/3289400432' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:10:17.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:16 smithi145 ceph-mon[26800]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:17.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:16 smithi145 ceph-mon[26800]: pgmap v286: 97 pgs: 97 active+clean; 322 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 11 KiB/s rd, 9.2 MiB/s wr, 798 op/s 2024-04-07T14:10:17.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:16 smithi145 ceph-mon[26800]: from='client.? 172.21.15.111:0/3289400432' entity='client.admin' cmd=[{"prefix": "health", "detail": "detail"}]: dispatch 2024-04-07T14:10:17.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 66361 18.88 MB/sec warmup 99 sec latency 144.282 ms 2024-04-07T14:10:17.917 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:17 smithi111 ceph-mon[24802]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:18.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:17 smithi145 ceph-mon[26800]: from='client.14666 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-07T14:10:18.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 66959 18.89 MB/sec warmup 100 sec latency 103.711 ms 2024-04-07T14:10:18.674 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:18 smithi111 ceph-mon[24802]: pgmap v287: 97 pgs: 97 active+clean; 327 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 826 op/s 2024-04-07T14:10:19.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:18 smithi145 ceph-mon[26800]: pgmap v287: 97 pgs: 97 active+clean; 327 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 16 KiB/s rd, 8.6 MiB/s wr, 826 op/s 2024-04-07T14:10:19.534 INFO:teuthology.orchestra.run.smithi111.stdout: 5 67608 18.94 MB/sec warmup 101 sec latency 26.161 ms 2024-04-07T14:10:20.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 68063 18.87 MB/sec warmup 102 sec latency 90.674 ms 2024-04-07T14:10:20.690 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:20 smithi111 ceph-mon[24802]: pgmap v288: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-04-07T14:10:21.022 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:20 smithi145 ceph-mon[26800]: pgmap v288: 97 pgs: 97 active+clean; 326 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 10 MiB/s wr, 1.00k op/s 2024-04-07T14:10:21.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 68684 18.77 MB/sec warmup 103 sec latency 24.423 ms 2024-04-07T14:10:22.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 69256 18.73 MB/sec warmup 104 sec latency 22.145 ms 2024-04-07T14:10:22.720 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:22 smithi111 ceph-mon[24802]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 769 op/s 2024-04-07T14:10:22.994 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:22 smithi145 ceph-mon[26800]: pgmap v289: 97 pgs: 97 active+clean; 330 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 KiB/s rd, 7.6 MiB/s wr, 769 op/s 2024-04-07T14:10:23.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 69957 18.71 MB/sec warmup 105 sec latency 34.531 ms 2024-04-07T14:10:24.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 70596 18.78 MB/sec warmup 106 sec latency 34.431 ms 2024-04-07T14:10:24.742 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:24 smithi145 ceph-mon[26800]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 832 op/s 2024-04-07T14:10:24.748 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:24 smithi111 ceph-mon[24802]: pgmap v290: 97 pgs: 97 active+clean; 332 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 8.5 MiB/s wr, 832 op/s 2024-04-07T14:10:25.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 71039 18.74 MB/sec warmup 107 sec latency 64.595 ms 2024-04-07T14:10:26.009 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:26.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:10:26.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:10:26.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:10:26.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: Standby manager daemon smithi145.qiunpb restarted 2024-04-07T14:10:26.010 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:25 smithi111 ceph-mon[24802]: Standby manager daemon smithi145.qiunpb started 2024-04-07T14:10:26.158 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:26.159 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:10:26.159 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-07T14:10:26.159 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-07T14:10:26.159 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: Standby manager daemon smithi145.qiunpb restarted 2024-04-07T14:10:26.159 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:25 smithi145 ceph-mon[26800]: Standby manager daemon smithi145.qiunpb started 2024-04-07T14:10:26.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 71549 18.68 MB/sec warmup 108 sec latency 46.024 ms 2024-04-07T14:10:26.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:26 smithi111 ceph-mon[24802]: pgmap v291: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.1 MiB/s wr, 779 op/s 2024-04-07T14:10:26.870 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:26 smithi111 ceph-mon[24802]: mgrmap e22: smithi111.rtrlso(active, since 7m), standbys: smithi145.qiunpb 2024-04-07T14:10:27.208 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:26 smithi145 ceph-mon[26800]: pgmap v291: 97 pgs: 97 active+clean; 335 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.7 KiB/s rd, 8.1 MiB/s wr, 779 op/s 2024-04-07T14:10:27.208 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:26 smithi145 ceph-mon[26800]: mgrmap e22: smithi111.rtrlso(active, since 7m), standbys: smithi145.qiunpb 2024-04-07T14:10:27.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 72108 18.61 MB/sec warmup 109 sec latency 20.605 ms 2024-04-07T14:10:28.025 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:27 smithi111 ceph-mon[24802]: Failed to load ceph-mgr modules: restful, mds_autoscaler, osd_perf_query, snap_schedule, dashboard, zabbix, balancer, influx, alerts, iostat, rgw, rbd_support, progress, pg_autoscaler, devicehealth, k8sevents, rook, diskprediction_local, mirroring, selftest, telegraf, test_orchestrator, localpool, crash, insights, orchestrator, osd_support, volumes, stats, cephadm, telemetry, prometheus, status, nfs 2024-04-07T14:10:28.250 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:27 smithi145 ceph-mon[26800]: Failed to load ceph-mgr modules: restful, mds_autoscaler, osd_perf_query, snap_schedule, dashboard, zabbix, balancer, influx, alerts, iostat, rgw, rbd_support, progress, pg_autoscaler, devicehealth, k8sevents, rook, diskprediction_local, mirroring, selftest, telegraf, test_orchestrator, localpool, crash, insights, orchestrator, osd_support, volumes, stats, cephadm, telemetry, prometheus, status, nfs 2024-04-07T14:10:28.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 72809 18.58 MB/sec warmup 110 sec latency 25.232 ms 2024-04-07T14:10:28.787 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:28 smithi111 ceph-mon[24802]: pgmap v292: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 787 op/s 2024-04-07T14:10:28.787 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:28 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:28 smithi145 ceph-mon[26800]: pgmap v292: 97 pgs: 97 active+clean; 337 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 14 KiB/s rd, 7.4 MiB/s wr, 787 op/s 2024-04-07T14:10:29.023 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:28 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:29.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 73582 18.57 MB/sec warmup 111 sec latency 29.293 ms 2024-04-07T14:10:30.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 74115 18.60 MB/sec warmup 112 sec latency 26.974 ms 2024-04-07T14:10:30.566 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:30 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:30.566 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:30 smithi111 ceph-mon[24802]: pgmap v293: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 904 op/s 2024-04-07T14:10:30.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:30 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:30.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:30 smithi145 ceph-mon[26800]: pgmap v293: 97 pgs: 97 active+clean; 341 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 8.8 MiB/s wr, 904 op/s 2024-04-07T14:10:31.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 74737 18.60 MB/sec warmup 113 sec latency 30.715 ms 2024-04-07T14:10:32.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 75107 18.54 MB/sec warmup 114 sec latency 39.734 ms 2024-04-07T14:10:32.806 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:32 smithi145 ceph-mon[26800]: pgmap v294: 97 pgs: 97 active+clean; 345 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 815 op/s 2024-04-07T14:10:32.873 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:32 smithi111 ceph-mon[24802]: pgmap v294: 97 pgs: 97 active+clean; 345 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 9.7 KiB/s rd, 7.4 MiB/s wr, 815 op/s 2024-04-07T14:10:33.537 INFO:teuthology.orchestra.run.smithi111.stdout: 5 75504 18.46 MB/sec warmup 115 sec latency 105.704 ms 2024-04-07T14:10:34.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.397 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: Upgrade: Need to upgrade myself (mgr.smithi111.rtrlso) 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: Failing over to other MGR 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr fail", "who": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: osdmap e70: 8 total, 8 up, 8 in 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: pgmap v296: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 847 op/s 2024-04-07T14:10:34.398 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:34 smithi111 ceph-mon[24802]: Standby manager daemon smithi111.rtrlso started 2024-04-07T14:10:34.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 76182 18.40 MB/sec warmup 116 sec latency 30.938 ms 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: Upgrade: Need to upgrade myself (mgr.smithi111.rtrlso) 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: Failing over to other MGR 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "mgr fail", "who": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: osdmap e70: 8 total, 8 up, 8 in 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: pgmap v296: 97 pgs: 97 active+clean; 339 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 12 KiB/s rd, 8.4 MiB/s wr, 847 op/s 2024-04-07T14:10:34.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:34 smithi145 ceph-mon[26800]: Standby manager daemon smithi111.rtrlso started 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi111.pvxsdw"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi145.fczrwb"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mgr metadata", "who": "smithi145.qiunpb", "id": "smithi145.qiunpb"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "mgr fail", "who": "smithi111.rtrlso"}]': finished 2024-04-07T14:10:35.402 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: mgrmap e23: smithi145.qiunpb(active, starting, since 1.0011s), standbys: smithi111.rtrlso 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:10:35.403 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:35 smithi111 ceph-mon[24802]: Manager daemon smithi145.qiunpb is now available 2024-04-07T14:10:35.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 76830 18.39 MB/sec warmup 117 sec latency 35.185 ms 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata", "id": "smithi111"}]: dispatch 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata", "id": "smithi145"}]: dispatch 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi111.pvxsdw"}]: dispatch 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata", "who": "foofs.smithi145.fczrwb"}]: dispatch 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mgr metadata", "who": "smithi145.qiunpb", "id": "smithi145.qiunpb"}]: dispatch 2024-04-07T14:10:35.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mgr metadata", "who": "smithi111.rtrlso", "id": "smithi111.rtrlso"}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.14200 172.21.15.111:0/3802129919' entity='mgr.smithi111.rtrlso' cmd='[{"prefix": "mgr fail", "who": "smithi111.rtrlso"}]': finished 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: mgrmap e23: smithi145.qiunpb(active, starting, since 1.0011s), standbys: smithi111.rtrlso 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: from='mgr.24413 172.21.15.145:0/1555993689' entity='mgr.smithi145.qiunpb' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-07T14:10:35.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:35 smithi145 ceph-mon[26800]: Manager daemon smithi145.qiunpb is now available 2024-04-07T14:10:36.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:36 smithi111 ceph-mon[24802]: mgrmap e24: smithi145.qiunpb(active, since 2s), standbys: smithi111.rtrlso 2024-04-07T14:10:36.408 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:36 smithi111 ceph-mon[24802]: pgmap v3: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:36.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 77413 18.38 MB/sec warmup 118 sec latency 35.252 ms 2024-04-07T14:10:36.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:36 smithi145 ceph-mon[26800]: mgrmap e24: smithi145.qiunpb(active, since 2s), standbys: smithi111.rtrlso 2024-04-07T14:10:36.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:36 smithi145 ceph-mon[26800]: pgmap v3: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:37.391 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:37 smithi111 ceph-mon[24802]: pgmap v4: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:37.392 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:37 smithi111 ceph-mon[24802]: Health check failed: 16 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY) 2024-04-07T14:10:37.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 77884 18.38 MB/sec warmup 119 sec latency 32.145 ms 2024-04-07T14:10:37.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:37 smithi145 ceph-mon[26800]: pgmap v4: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:37.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:37 smithi145 ceph-mon[26800]: Health check failed: 16 mgr modules have failed dependencies (MGR_MODULE_DEPENDENCY) 2024-04-07T14:10:39.411 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:39 smithi111 ceph-mon[24802]: pgmap v5: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:39.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 79001 16.58 MB/sec execute 1 sec latency 27.870 ms 2024-04-07T14:10:39.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:39 smithi145 ceph-mon[26800]: pgmap v5: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:40.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 79794 14.55 MB/sec execute 2 sec latency 19.869 ms 2024-04-07T14:10:41.422 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:41 smithi111 ceph-mon[24802]: pgmap v6: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:41.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 80567 16.48 MB/sec execute 3 sec latency 29.561 ms 2024-04-07T14:10:41.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:41 smithi145 ceph-mon[26800]: pgmap v6: 97 pgs: 97 active+clean; 333 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail 2024-04-07T14:10:42.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 81150 17.14 MB/sec execute 4 sec latency 33.517 ms 2024-04-07T14:10:43.444 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:43 smithi111 ceph-mon[24802]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.4 MiB/s wr, 589 op/s 2024-04-07T14:10:43.444 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:43 smithi111 ceph-mon[24802]: Standby manager daemon smithi111.rtrlso restarted 2024-04-07T14:10:43.445 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:43 smithi111 ceph-mon[24802]: Standby manager daemon smithi111.rtrlso started 2024-04-07T14:10:43.445 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:43 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.rtrlso/crt"}]: dispatch 2024-04-07T14:10:43.445 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:43 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T14:10:43.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 81587 17.17 MB/sec execute 5 sec latency 57.837 ms 2024-04-07T14:10:43.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:43 smithi145 ceph-mon[26800]: pgmap v7: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.0 KiB/s rd, 6.4 MiB/s wr, 589 op/s 2024-04-07T14:10:43.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:43 smithi145 ceph-mon[26800]: Standby manager daemon smithi111.rtrlso restarted 2024-04-07T14:10:43.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:43 smithi145 ceph-mon[26800]: Standby manager daemon smithi111.rtrlso started 2024-04-07T14:10:43.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:43 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.rtrlso/crt"}]: dispatch 2024-04-07T14:10:43.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:43 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-07T14:10:44.472 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:44 smithi111 ceph-mon[24802]: mgrmap e25: smithi145.qiunpb(active, since 10s), standbys: smithi111.rtrlso 2024-04-07T14:10:44.472 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:44 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.rtrlso/key"}]: dispatch 2024-04-07T14:10:44.472 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:44 smithi111 ceph-mon[24802]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T14:10:44.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 82118 16.87 MB/sec execute 6 sec latency 40.989 ms 2024-04-07T14:10:44.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:44 smithi145 ceph-mon[26800]: mgrmap e25: smithi145.qiunpb(active, since 10s), standbys: smithi111.rtrlso 2024-04-07T14:10:44.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:44 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi111.rtrlso/key"}]: dispatch 2024-04-07T14:10:44.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:44 smithi145 ceph-mon[26800]: from='mgr.? 172.21.15.111:0/2256843880' entity='mgr.smithi111.rtrlso' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-07T14:10:45.492 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:45 smithi111 ceph-mon[24802]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 458 op/s 2024-04-07T14:10:45.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 82529 15.62 MB/sec execute 7 sec latency 31.703 ms 2024-04-07T14:10:45.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:45 smithi145 ceph-mon[26800]: pgmap v8: 97 pgs: 97 active+clean; 340 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.5 KiB/s rd, 5.0 MiB/s wr, 458 op/s 2024-04-07T14:10:46.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 83241 15.00 MB/sec execute 8 sec latency 31.189 ms 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:no valid command found; 10 closest matches: 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg stat 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg getmap 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_json [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_pools_json 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-pool [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-primary [] [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-osd [] [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls [] [...] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_stuck [...] [] 2024-04-07T14:10:46.997 INFO:teuthology.orchestra.run.smithi111.stderr:Error EINVAL: invalid command 2024-04-07T14:10:47.508 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:47 smithi111 ceph-mon[24802]: pgmap v9: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 731 op/s 2024-04-07T14:10:47.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 84017 15.91 MB/sec execute 9 sec latency 22.782 ms 2024-04-07T14:10:47.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:47 smithi145 ceph-mon[26800]: pgmap v9: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 KiB/s rd, 8.1 MiB/s wr, 731 op/s 2024-04-07T14:10:48.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 84638 16.34 MB/sec execute 10 sec latency 28.695 ms 2024-04-07T14:10:48.908 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps' 2024-04-07T14:10:49.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 85098 16.48 MB/sec execute 11 sec latency 26.017 ms 2024-04-07T14:10:49.544 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:49 smithi111 ceph-mon[24802]: pgmap v10: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 667 op/s 2024-04-07T14:10:49.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:49 smithi145 ceph-mon[26800]: pgmap v10: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 667 op/s 2024-04-07T14:10:50.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 85683 16.40 MB/sec execute 12 sec latency 29.653 ms 2024-04-07T14:10:51.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 86310 16.10 MB/sec execute 13 sec latency 16.924 ms 2024-04-07T14:10:51.562 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:51 smithi111 ceph-mon[24802]: pgmap v11: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 667 op/s 2024-04-07T14:10:51.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:51 smithi145 ceph-mon[26800]: pgmap v11: 97 pgs: 97 active+clean; 336 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 9.5 KiB/s rd, 7.4 MiB/s wr, 667 op/s 2024-04-07T14:10:52.386 INFO:teuthology.orchestra.run.smithi111.stderr:no valid command found; 10 closest matches: 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg stat 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg getmap 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_json [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_pools_json 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-pool [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-primary [] [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls-by-osd [] [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg ls [] [...] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:pg dump_stuck [...] [] 2024-04-07T14:10:52.387 INFO:teuthology.orchestra.run.smithi111.stderr:Error EINVAL: invalid command 2024-04-07T14:10:52.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 86922 15.96 MB/sec execute 14 sec latency 26.059 ms 2024-04-07T14:10:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-04-07T14:10:53.362 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 1459, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps'" 2024-04-07T14:10:53.568 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=76b7dc0fd9b34ae88efcb444ac92ff2d Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 1459, in shell _shell( File "/home/teuthworker/src/git.ceph.com_ceph_e5c885fe95ee982ff7b45214daa61b1a6d09d57d/qa/tasks/cephadm.py", line 110, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi111 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c 'ceph orch ps'" 2024-04-07T14:10:53.572 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-07T14:10:53.582 INFO:tasks.cephadm:Teardown begin 2024-04-07T14:10:53.583 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:10:53.589 INFO:teuthology.orchestra.run.smithi111.stdout: 5 87572 15.91 MB/sec execute 15 sec latency 31.706 ms 2024-04-07T14:10:53.589 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:53 smithi111 ceph-mon[24802]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-07T14:10:53.635 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:10:53.666 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-07T14:10:53.667 DEBUG:teuthology.orchestra.run.smithi111:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-07T14:10:53.697 DEBUG:teuthology.orchestra.run.smithi145:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-07T14:10:53.722 INFO:tasks.cephadm:Stopping all daemons... 2024-04-07T14:10:53.723 INFO:tasks.cephadm.mon.smithi111:Stopping mon.smithi111... 2024-04-07T14:10:53.723 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111 2024-04-07T14:10:53.772 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:53 smithi145 ceph-mon[26800]: pgmap v12: 97 pgs: 97 active+clean; 333 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 15 KiB/s rd, 11 MiB/s wr, 1.03k op/s 2024-04-07T14:10:53.827 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:53 smithi111 systemd[1]: Stopping Ceph mon.smithi111 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:10:53.963 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:53 smithi111 bash[57291]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon.smithi111" found: no such container 2024-04-07T14:10:54.079 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:54 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111[24798]: 2024-04-07T14:10:54.066+0000 7f413177d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi111 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:10:54.080 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:54 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111[24798]: 2024-04-07T14:10:54.066+0000 7f413177d700 -1 mon.smithi111@0(leader) e2 *** Got Signal Terminated *** 2024-04-07T14:10:54.331 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:54 smithi111 podman[57300]: 2024-04-07 14:10:54.087909262 +0000 UTC m=+0.237696353 container died 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, ceph=True, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.openshift.expose-services=, name=centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, architecture=x86_64, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, release=754, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8) 2024-04-07T14:10:54.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 88341 16.63 MB/sec execute 16 sec latency 26.621 ms 2024-04-07T14:10:54.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:54 smithi111 podman[57300]: 2024-04-07 14:10:54.734349647 +0000 UTC m=+0.884136731 container cleanup 382c73ff91e29d313c5815ddab1e1d758ab47288adadbf4f14e17cd722e7b4be (image=quay.io/ceph/ceph:v17.2.0, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111, com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.openshift.expose-services=, RELEASE=HEAD, name=centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, ceph=True, version=8, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T14:10:54.840 INFO:journalctl@ceph.mon.smithi111.smithi111.stdout:Apr 07 14:10:54 smithi111 bash[57300]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi111 2024-04-07T14:10:55.240 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi111.service' 2024-04-07T14:10:55.307 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:10:55.307 INFO:tasks.cephadm.mon.smithi111:Stopped mon.smithi111 2024-04-07T14:10:55.307 INFO:tasks.cephadm.mon.smithi145:Stopping mon.smithi145... 2024-04-07T14:10:55.307 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi145 2024-04-07T14:10:55.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89102 17.13 MB/sec execute 17 sec latency 26.338 ms 2024-04-07T14:10:55.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:55 smithi145 systemd[1]: Stopping Ceph mon.smithi145 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:10:55.773 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:55 smithi145 bash[44029]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon.smithi145" found: no such container 2024-04-07T14:10:55.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145[26784]: 2024-04-07T14:10:55.547+0000 7f3cee441700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi145 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:10:55.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145[26784]: 2024-04-07T14:10:55.547+0000 7f3cee441700 -1 mon.smithi145@1(peon) e2 *** Got Signal Terminated *** 2024-04-07T14:10:55.774 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:55 smithi145 podman[44039]: 2024-04-07 14:10:55.56809982 +0000 UTC m=+0.169974352 container died c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, vendor=Red Hat, Inc., name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-type=git, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.component=centos-stream-container, distribution-scope=public, ceph=True, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T14:10:56.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:56 smithi145 podman[44039]: 2024-04-07 14:10:56.080109386 +0000 UTC m=+0.681983915 container cleanup c8daa5dcac8e73b8b84006ce42277d770a62443422c5b04819b4f36f97f8f146 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, distribution-scope=public, name=centos-stream, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.buildah.version=1.19.8, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-07T14:10:56.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:56 smithi145 bash[44039]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon-smithi145 2024-04-07T14:10:56.273 INFO:journalctl@ceph.mon.smithi145.smithi145.stdout:Apr 07 14:10:56 smithi145 bash[44066]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-mon.smithi145" found: no such container 2024-04-07T14:10:56.428 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@mon.smithi145.service' 2024-04-07T14:10:56.472 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:10:56.472 INFO:tasks.cephadm.mon.smithi145:Stopped mon.smithi145 2024-04-07T14:10:56.473 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-07T14:10:56.473 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.0 2024-04-07T14:10:56.538 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 16.19 MB/sec execute 18 sec latency 969.097 ms 2024-04-07T14:10:56.976 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:10:56 smithi111 systemd[1]: Stopping Ceph osd.0 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:10:56.977 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:10:56 smithi111 bash[57377]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.0" found: no such container 2024-04-07T14:10:56.977 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:10:56 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0[35234]: 2024-04-07T14:10:56.701+0000 7f01d4aaf700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:10:56.977 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:10:56 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0[35234]: 2024-04-07T14:10:56.701+0000 7f01d4aaf700 -1 osd.0 70 *** Got signal Terminated *** 2024-04-07T14:10:56.977 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:10:56 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0[35234]: 2024-04-07T14:10:56.701+0000 7f01d4aaf700 -1 osd.0 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:10:57.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 15.34 MB/sec execute 19 sec latency 1969.154 ms 2024-04-07T14:10:58.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 14.57 MB/sec execute 20 sec latency 2969.216 ms 2024-04-07T14:10:59.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 13.88 MB/sec execute 21 sec latency 3969.261 ms 2024-04-07T14:11:00.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 13.25 MB/sec execute 22 sec latency 4969.340 ms 2024-04-07T14:11:01.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 12.67 MB/sec execute 23 sec latency 5969.394 ms 2024-04-07T14:11:02.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 12.14 MB/sec execute 24 sec latency 6969.445 ms 2024-04-07T14:11:02.581 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:02 smithi111 podman[57386]: 2024-04-07 14:11:02.309565546 +0000 UTC m=+5.735469279 container died ea5afbacec5e0ddeecfaf7e7aae6b46d122d01caf51eea68e500fdde4f3777f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0, RELEASE=HEAD, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, release=754, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, version=8, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-07T14:11:02.972 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:02 smithi111 podman[57386]: 2024-04-07 14:11:02.705945896 +0000 UTC m=+6.131849635 container cleanup ea5afbacec5e0ddeecfaf7e7aae6b46d122d01caf51eea68e500fdde4f3777f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, maintainer=Guillaume Abrioux , io.openshift.expose-services=, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, name=centos-stream, io.buildah.version=1.19.8, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, release=754) 2024-04-07T14:11:02.972 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:02 smithi111 bash[57386]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0 2024-04-07T14:11:02.973 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:02 smithi111 bash[57417]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.0" found: no such container 2024-04-07T14:11:03.226 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:02 smithi111 podman[57405]: 2024-04-07 14:11:02.973553954 +0000 UTC m=+0.654852302 container remove ea5afbacec5e0ddeecfaf7e7aae6b46d122d01caf51eea68e500fdde4f3777f7 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, release=754, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, name=centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8) 2024-04-07T14:11:03.226 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.115389424 +0000 UTC m=+0.020112560 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:03.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 11.66 MB/sec execute 25 sec latency 7969.500 ms 2024-04-07T14:11:03.551 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.274562555 +0000 UTC m=+0.179285689 container create cdad92fbe4f6dedc169e2106d83f50c5e8be22db3af7ec16fdb217f19d581baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=-17.2.0, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, vcs-type=git, distribution-scope=public, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, ceph=True, architecture=x86_64, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, name=centos-stream, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=) 2024-04-07T14:11:03.551 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.552685758 +0000 UTC m=+0.457408900 container init cdad92fbe4f6dedc169e2106d83f50c5e8be22db3af7ec16fdb217f19d581baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0-deactivate, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc., GIT_CLEAN=True, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_BRANCH=HEAD, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, io.buildah.version=1.19.8, release=754, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, vcs-type=git) 2024-04-07T14:11:03.928 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.562837759 +0000 UTC m=+0.467560903 container start cdad92fbe4f6dedc169e2106d83f50c5e8be22db3af7ec16fdb217f19d581baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0-deactivate, distribution-scope=public, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, ceph=True, vendor=Red Hat, Inc., io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, version=8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, name=centos-stream, maintainer=Guillaume Abrioux , architecture=x86_64) 2024-04-07T14:11:03.928 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.652610565 +0000 UTC m=+0.557333699 container attach cdad92fbe4f6dedc169e2106d83f50c5e8be22db3af7ec16fdb217f19d581baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0-deactivate, CEPH_POINT_RELEASE=-17.2.0, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , version=8, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-type=git, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, vendor=Red Hat, Inc., name=centos-stream, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64) 2024-04-07T14:11:04.226 INFO:journalctl@ceph.osd.0.smithi111.stdout:Apr 07 14:11:03 smithi111 podman[57481]: 2024-04-07 14:11:03.9291321 +0000 UTC m=+0.833855236 container died cdad92fbe4f6dedc169e2106d83f50c5e8be22db3af7ec16fdb217f19d581baa (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-0-deactivate, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, version=8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, release=754, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, ceph=True, vendor=Red Hat, Inc., vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, name=centos-stream, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD) 2024-04-07T14:11:04.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 11.21 MB/sec execute 26 sec latency 8969.530 ms 2024-04-07T14:11:04.558 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.0.service' 2024-04-07T14:11:04.599 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:04.599 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-07T14:11:04.599 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-07T14:11:04.599 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.1 2024-04-07T14:11:04.844 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:04 smithi111 systemd[1]: Stopping Ceph osd.1 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:04.845 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:04 smithi111 bash[57560]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.1" found: no such container 2024-04-07T14:11:05.226 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:04 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1[39298]: 2024-04-07T14:11:04.844+0000 7f51b89ef700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:05.226 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:04 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1[39298]: 2024-04-07T14:11:04.844+0000 7f51b89ef700 -1 osd.1 70 *** Got signal Terminated *** 2024-04-07T14:11:05.226 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:04 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1[39298]: 2024-04-07T14:11:04.844+0000 7f51b89ef700 -1 osd.1 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:05.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 10.79 MB/sec execute 27 sec latency 9969.570 ms 2024-04-07T14:11:06.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 10.41 MB/sec execute 28 sec latency 10969.636 ms 2024-04-07T14:11:07.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 10.05 MB/sec execute 29 sec latency 11969.693 ms 2024-04-07T14:11:08.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 9.71 MB/sec execute 30 sec latency 12969.744 ms 2024-04-07T14:11:09.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 9.40 MB/sec execute 31 sec latency 13969.796 ms 2024-04-07T14:11:10.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 9.11 MB/sec execute 32 sec latency 14969.835 ms 2024-04-07T14:11:10.677 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:10 smithi111 podman[57570]: 2024-04-07 14:11:10.413185356 +0000 UTC m=+5.671643096 container died 1da0bd36d060dacf84ccbd918368c576955ea9a904a8d8b622172ca4ee4a4922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1, architecture=x86_64, distribution-scope=public, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, vcs-type=git, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, version=8, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, maintainer=Guillaume Abrioux , io.openshift.expose-services=, io.buildah.version=1.19.8, vendor=Red Hat, Inc.) 2024-04-07T14:11:10.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:10 smithi111 podman[57570]: 2024-04-07 14:11:10.766212902 +0000 UTC m=+6.024670641 container cleanup 1da0bd36d060dacf84ccbd918368c576955ea9a904a8d8b622172ca4ee4a4922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, RELEASE=HEAD, io.buildah.version=1.19.8, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-type=git, version=8, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T14:11:10.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:10 smithi111 bash[57570]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1 2024-04-07T14:11:10.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:10 smithi111 bash[57602]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.1" found: no such container 2024-04-07T14:11:11.368 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57589]: 2024-04-07 14:11:11.034443111 +0000 UTC m=+0.618684301 container remove 1da0bd36d060dacf84ccbd918368c576955ea9a904a8d8b622172ca4ee4a4922 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, version=8, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, ceph=True, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0) 2024-04-07T14:11:11.368 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57665]: 2024-04-07 14:11:11.127021267 +0000 UTC m=+0.018632406 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:11.368 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57665]: 2024-04-07 14:11:11.235197041 +0000 UTC m=+0.126808165 container create 09bd904f706e3935c1a016f35056a46e5b22f87c6ba9c0a15b36e77aa6a69719 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.openshift.expose-services=, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, io.buildah.version=1.19.8, version=8, architecture=x86_64, vcs-type=git, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-04-07T14:11:11.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 8.83 MB/sec execute 33 sec latency 15969.863 ms 2024-04-07T14:11:11.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57665]: 2024-04-07 14:11:11.679445351 +0000 UTC m=+0.571056475 container init 09bd904f706e3935c1a016f35056a46e5b22f87c6ba9c0a15b36e77aa6a69719 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1-deactivate, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, RELEASE=HEAD, version=8, release=754, GIT_CLEAN=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream) 2024-04-07T14:11:11.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57665]: 2024-04-07 14:11:11.682847287 +0000 UTC m=+0.574458419 container start 09bd904f706e3935c1a016f35056a46e5b22f87c6ba9c0a15b36e77aa6a69719 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1-deactivate, vendor=Red Hat, Inc., architecture=x86_64, io.k8s.display-name=CentOS Stream 8, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_BRANCH=HEAD, GIT_CLEAN=True, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, ceph=True, distribution-scope=public, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, version=8, io.openshift.expose-services=, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T14:11:11.976 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:11 smithi111 podman[57665]: 2024-04-07 14:11:11.76275265 +0000 UTC m=+0.654363775 container attach 09bd904f706e3935c1a016f35056a46e5b22f87c6ba9c0a15b36e77aa6a69719 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1-deactivate, vendor=Red Hat, Inc., io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, io.openshift.expose-services=, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, ceph=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, release=754, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T14:11:12.331 INFO:journalctl@ceph.osd.1.smithi111.stdout:Apr 07 14:11:12 smithi111 podman[57665]: 2024-04-07 14:11:12.018012216 +0000 UTC m=+0.909623347 container died 09bd904f706e3935c1a016f35056a46e5b22f87c6ba9c0a15b36e77aa6a69719 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-1-deactivate, io.k8s.display-name=CentOS Stream 8, release=754, vcs-type=git, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, architecture=x86_64, io.openshift.tags=base centos centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.buildah.version=1.19.8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, name=centos-stream, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly.) 2024-04-07T14:11:12.539 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 8.57 MB/sec execute 34 sec latency 16969.931 ms 2024-04-07T14:11:12.694 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.1.service' 2024-04-07T14:11:12.733 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:12.733 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-07T14:11:12.734 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-07T14:11:12.734 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.2 2024-04-07T14:11:12.976 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:12 smithi111 systemd[1]: Stopping Ceph osd.2 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:12.976 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:12 smithi111 bash[57742]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.2" found: no such container 2024-04-07T14:11:13.476 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:12 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2[43000]: 2024-04-07T14:11:12.988+0000 7fb332942700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:13.477 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:12 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2[43000]: 2024-04-07T14:11:12.988+0000 7fb332942700 -1 osd.2 70 *** Got signal Terminated *** 2024-04-07T14:11:13.477 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:12 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2[43000]: 2024-04-07T14:11:12.988+0000 7fb332942700 -1 osd.2 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:13.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 8.33 MB/sec execute 35 sec latency 17969.958 ms 2024-04-07T14:11:14.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 8.10 MB/sec execute 36 sec latency 18970.022 ms 2024-04-07T14:11:15.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 7.88 MB/sec execute 37 sec latency 19970.071 ms 2024-04-07T14:11:16.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 7.67 MB/sec execute 38 sec latency 20970.124 ms 2024-04-07T14:11:17.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 7.47 MB/sec execute 39 sec latency 21970.186 ms 2024-04-07T14:11:18.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 7.29 MB/sec execute 40 sec latency 22970.210 ms 2024-04-07T14:11:18.917 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:18 smithi111 podman[57755]: 2024-04-07 14:11:18.572660192 +0000 UTC m=+5.694210306 container died c6ab9df37c927f2fe9d4b33c6e652550c62315fc8bd43c94506c2762a195679e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vendor=Red Hat, Inc., distribution-scope=public, ceph=True, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git) 2024-04-07T14:11:19.226 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:18 smithi111 podman[57755]: 2024-04-07 14:11:18.918468401 +0000 UTC m=+6.040018513 container cleanup c6ab9df37c927f2fe9d4b33c6e652550c62315fc8bd43c94506c2762a195679e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2, architecture=x86_64, version=8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, name=centos-stream, ceph=True, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, vendor=Red Hat, Inc., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T14:11:19.226 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:18 smithi111 bash[57755]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2 2024-04-07T14:11:19.226 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57774]: 2024-04-07 14:11:19.077928713 +0000 UTC m=+0.500345315 container remove c6ab9df37c927f2fe9d4b33c6e652550c62315fc8bd43c94506c2762a195679e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2, io.openshift.expose-services=, release=754, RELEASE=HEAD, architecture=x86_64, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, version=8, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, vcs-type=git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, ceph=True, distribution-scope=public, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream) 2024-04-07T14:11:19.227 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 bash[57788]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.2" found: no such container 2024-04-07T14:11:19.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 7.11 MB/sec execute 41 sec latency 23970.234 ms 2024-04-07T14:11:19.647 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57850]: 2024-04-07 14:11:19.285212146 +0000 UTC m=+0.018637227 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:19.647 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57850]: 2024-04-07 14:11:19.386969698 +0000 UTC m=+0.120394761 container create 8cf016adc9e4c4c0943b03b5781e6225bd179217df431f2525f63dce6b6f3d5d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2-deactivate, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, vendor=Red Hat, Inc., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, RELEASE=HEAD, ceph=True, io.openshift.tags=base centos centos-stream) 2024-04-07T14:11:19.647 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57850]: 2024-04-07 14:11:19.648143568 +0000 UTC m=+0.381568650 container init 8cf016adc9e4c4c0943b03b5781e6225bd179217df431f2525f63dce6b6f3d5d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2-deactivate, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-type=git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.buildah.version=1.19.8, version=8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.expose-services=, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T14:11:19.976 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57850]: 2024-04-07 14:11:19.653847286 +0000 UTC m=+0.387272352 container start 8cf016adc9e4c4c0943b03b5781e6225bd179217df431f2525f63dce6b6f3d5d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2-deactivate, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.buildah.version=1.19.8, distribution-scope=public, ceph=True, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, io.openshift.expose-services=, RELEASE=HEAD, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870) 2024-04-07T14:11:19.977 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:19 smithi111 podman[57850]: 2024-04-07 14:11:19.731307999 +0000 UTC m=+0.464733077 container attach 8cf016adc9e4c4c0943b03b5781e6225bd179217df431f2525f63dce6b6f3d5d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , release=754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., architecture=x86_64, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, distribution-scope=public, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., vcs-type=git, version=8, name=centos-stream, com.redhat.component=centos-stream-container, build-date=2022-05-03T08:36:31.336870, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0) 2024-04-07T14:11:20.291 INFO:journalctl@ceph.osd.2.smithi111.stdout:Apr 07 14:11:20 smithi111 podman[57850]: 2024-04-07 14:11:20.004649135 +0000 UTC m=+0.738074201 container died 8cf016adc9e4c4c0943b03b5781e6225bd179217df431f2525f63dce6b6f3d5d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-2-deactivate, maintainer=Guillaume Abrioux , io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.component=centos-stream-container, RELEASE=HEAD, release=754, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_REPO=https://github.com/ceph/ceph-container.git, name=centos-stream, io.openshift.expose-services=, distribution-scope=public, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, ceph=True, architecture=x86_64, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, vendor=Red Hat, Inc.) 2024-04-07T14:11:20.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.94 MB/sec execute 42 sec latency 24970.286 ms 2024-04-07T14:11:20.642 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.2.service' 2024-04-07T14:11:20.682 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:20.683 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-07T14:11:20.683 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-07T14:11:20.683 DEBUG:teuthology.orchestra.run.smithi111:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.3 2024-04-07T14:11:20.943 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:20 smithi111 systemd[1]: Stopping Ceph osd.3 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:20.944 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:20 smithi111 bash[57928]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.3" found: no such container 2024-04-07T14:11:21.226 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:20 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3[46261]: 2024-04-07T14:11:20.943+0000 7f44485a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:21.226 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:20 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3[46261]: 2024-04-07T14:11:20.943+0000 7f44485a7700 -1 osd.3 70 *** Got signal Terminated *** 2024-04-07T14:11:21.226 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:20 smithi111 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3[46261]: 2024-04-07T14:11:20.943+0000 7f44485a7700 -1 osd.3 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:21.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.78 MB/sec execute 43 sec latency 25970.351 ms 2024-04-07T14:11:22.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.62 MB/sec execute 44 sec latency 26970.409 ms 2024-04-07T14:11:23.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:22 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:22.748+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:23.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:23 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:23.269+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:23.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.48 MB/sec execute 45 sec latency 27970.459 ms 2024-04-07T14:11:24.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:23 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:23.745+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:24.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.34 MB/sec execute 46 sec latency 28970.522 ms 2024-04-07T14:11:24.696 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:24 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:24.309+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:25.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:24 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:24.697+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:25.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.20 MB/sec execute 47 sec latency 29970.550 ms 2024-04-07T14:11:25.658 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:25 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:25.274+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:26.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:25 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:25.724+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:26.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:25 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:25.744+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:26.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:25 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:25.658+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:26.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:26 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:26.225+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:26.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 6.07 MB/sec execute 48 sec latency 30970.578 ms 2024-04-07T14:11:26.678 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:26 smithi111 podman[57941]: 2024-04-07 14:11:26.427992491 +0000 UTC m=+5.600630470 container died 195f5ab7aba6a6302cd781bc7775353887514cbcff448cc3d9716ea646289514 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3, vcs-type=git, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, io.openshift.tags=base centos centos-stream, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, name=centos-stream, io.buildah.version=1.19.8, io.openshift.expose-services=, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., GIT_CLEAN=True, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, distribution-scope=public, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T14:11:26.976 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:26 smithi111 podman[57941]: 2024-04-07 14:11:26.76992285 +0000 UTC m=+5.942560828 container cleanup 195f5ab7aba6a6302cd781bc7775353887514cbcff448cc3d9716ea646289514 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.tags=base centos centos-stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , RELEASE=HEAD, vcs-type=git, io.buildah.version=1.19.8) 2024-04-07T14:11:26.976 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:26 smithi111 bash[57941]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3 2024-04-07T14:11:26.976 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:26 smithi111 bash[57972]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.3" found: no such container 2024-04-07T14:11:27.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:26 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:26.789+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:27.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:26 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:26.692+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:27.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:26 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:26.670+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:27.404 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[57960]: 2024-04-07 14:11:27.0797258 +0000 UTC m=+0.656063499 container remove 195f5ab7aba6a6302cd781bc7775353887514cbcff448cc3d9716ea646289514 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, name=centos-stream, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, io.buildah.version=1.19.8, release=754, ceph=True, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, GIT_CLEAN=True, vcs-type=git, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc., distribution-scope=public, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T14:11:27.404 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.170595666 +0000 UTC m=+0.018588625 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:27.405 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.272166919 +0000 UTC m=+0.120159888 container create da5fa31749758ed157814b082d9cb8a8badcf1132c1d599f9dfc80976ea4f5cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3-deactivate, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, io.buildah.version=1.19.8, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, distribution-scope=public, name=centos-stream, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, architecture=x86_64, release=754, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image) 2024-04-07T14:11:27.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:27.225+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:27.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.95 MB/sec execute 49 sec latency 31970.611 ms 2024-04-07T14:11:27.726 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.500682102 +0000 UTC m=+0.348675062 container init da5fa31749758ed157814b082d9cb8a8badcf1132c1d599f9dfc80976ea4f5cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3-deactivate, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, GIT_BRANCH=HEAD, distribution-scope=public, name=centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , vcs-type=git, ceph=True, architecture=x86_64, io.openshift.expose-services=, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, release=754) 2024-04-07T14:11:27.726 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.507969166 +0000 UTC m=+0.355962125 container start da5fa31749758ed157814b082d9cb8a8badcf1132c1d599f9dfc80976ea4f5cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3-deactivate, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, distribution-scope=public, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.k8s.display-name=CentOS Stream 8, version=8, io.buildah.version=1.19.8, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T14:11:27.726 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.600788313 +0000 UTC m=+0.448781286 container attach da5fa31749758ed157814b082d9cb8a8badcf1132c1d599f9dfc80976ea4f5cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3-deactivate, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git, ceph=True, version=8, io.buildah.version=1.19.8, io.k8s.display-name=CentOS Stream 8, distribution-scope=public) 2024-04-07T14:11:28.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:27.736+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:28.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:27.790+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:28.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:27 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:27.674+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:28.144 INFO:journalctl@ceph.osd.3.smithi111.stdout:Apr 07 14:11:27 smithi111 podman[58031]: 2024-04-07 14:11:27.858871426 +0000 UTC m=+0.706864377 container died da5fa31749758ed157814b082d9cb8a8badcf1132c1d599f9dfc80976ea4f5cf (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-3-deactivate, ceph=True, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, RELEASE=HEAD, io.k8s.display-name=CentOS Stream 8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-type=git, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vendor=Red Hat, Inc., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_BRANCH=HEAD, name=centos-stream, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T14:11:28.436 DEBUG:teuthology.orchestra.run.smithi111:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.3.service' 2024-04-07T14:11:28.476 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:28.476 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-07T14:11:28.476 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-07T14:11:28.477 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.4 2024-04-07T14:11:28.505 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:28.240+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:28.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.83 MB/sec execute 50 sec latency 32970.642 ms 2024-04-07T14:11:28.773 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:28.704+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:28.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:28.679+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:28.773 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 systemd[1]: Stopping Ceph osd.4 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:28.774 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 bash[44137]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.4" found: no such container 2024-04-07T14:11:28.774 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:28.706+0000 7f9981830700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:28.774 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:28.706+0000 7f9981830700 -1 osd.4 70 *** Got signal Terminated *** 2024-04-07T14:11:28.774 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:28.706+0000 7f9981830700 -1 osd.4 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:29.212 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:28 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:28.807+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:29.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:29 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:29.212+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:29.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.71 MB/sec execute 51 sec latency 33970.704 ms 2024-04-07T14:11:30.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:29 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:29.644+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:30.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:29 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:29.811+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:30.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:29 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:29.672+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:30.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:30 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:30.181+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:30.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.60 MB/sec execute 52 sec latency 34970.727 ms 2024-04-07T14:11:31.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:30 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:30.654+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:31.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:30 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:30.821+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:31.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:30 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:30.676+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:31.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:31.135+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:31.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.50 MB/sec execute 53 sec latency 35970.786 ms 2024-04-07T14:11:31.869 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:31.637+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:31.870 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:31.611+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:31.870 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:31.611+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:32.124 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:31.869+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:32.124 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:31 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:31.870+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:32.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:32.124+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:32.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.40 MB/sec execute 54 sec latency 36970.833 ms 2024-04-07T14:11:32.899 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:32.616+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:32.900 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:32.630+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:32.900 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:32.630+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:33.273 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:33.075+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:57.381242+0000 front 2024-04-07T14:10:57.381200+0000 (oldest deadline 2024-04-07T14:11:22.581859+0000) 2024-04-07T14:11:33.273 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4[31036]: 2024-04-07T14:11:33.075+0000 7f997e1cb700 -1 osd.4 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:09.583139+0000 front 2024-04-07T14:11:09.583154+0000 (oldest deadline 2024-04-07T14:11:32.483268+0000) 2024-04-07T14:11:33.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:32.899+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:33.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:32 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:32.899+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:33.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.30 MB/sec execute 55 sec latency 37970.878 ms 2024-04-07T14:11:33.894 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:33.639+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:33.894 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:33.639+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:33.894 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:33.594+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:33.894 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:33.594+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:34.271 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:33.893+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:34.271 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:33 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:33.893+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:34.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:34 smithi145 podman[44150]: 2024-04-07 14:11:34.272194916 +0000 UTC m=+5.707014226 container died 06ffbb78268c38f585fdd89bf374272b54eb1946a5a69d70629bbab2adf4dae2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4, com.redhat.component=centos-stream-container, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.tags=base centos centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.buildah.version=1.19.8, vendor=Red Hat, Inc., RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, build-date=2022-05-03T08:36:31.336870, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T14:11:34.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.20 MB/sec execute 56 sec latency 38970.920 ms 2024-04-07T14:11:34.808 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:34.634+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:34.808 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:34.634+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:34.809 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:34.557+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:34.809 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:34.557+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:34.809 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:34 smithi145 podman[44150]: 2024-04-07 14:11:34.766061728 +0000 UTC m=+6.200881031 container cleanup 06ffbb78268c38f585fdd89bf374272b54eb1946a5a69d70629bbab2adf4dae2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4, ceph=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, CEPH_POINT_RELEASE=-17.2.0, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, GIT_CLEAN=True, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, name=centos-stream, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, RELEASE=HEAD, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, release=754, vcs-type=git, io.buildah.version=1.19.8, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux ) 2024-04-07T14:11:34.809 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:34 smithi145 bash[44150]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4 2024-04-07T14:11:34.809 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:34 smithi145 bash[44192]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.4" found: no such container 2024-04-07T14:11:35.250 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44178]: 2024-04-07 14:11:35.025923137 +0000 UTC m=+0.755512960 container remove 06ffbb78268c38f585fdd89bf374272b54eb1946a5a69d70629bbab2adf4dae2 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, vcs-type=git, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, name=centos-stream, GIT_CLEAN=True, distribution-scope=public, ceph=True, io.buildah.version=1.19.8, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, io.k8s.display-name=CentOS Stream 8, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container) 2024-04-07T14:11:35.250 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:34.918+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:35.250 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:34 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:34.918+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:35.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.154841135 +0000 UTC m=+0.019068392 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:35.523 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.277099968 +0000 UTC m=+0.141327224 container create ec448797ff779068fc2fc50ef57022b9286175e8843fbbb7cbbf0407058c0af6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, release=754, GIT_BRANCH=HEAD, distribution-scope=public, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, vendor=Red Hat, Inc., com.redhat.component=centos-stream-container, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, io.buildah.version=1.19.8, io.openshift.expose-services=, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, ceph=True, GIT_REPO=https://github.com/ceph/ceph-container.git, build-date=2022-05-03T08:36:31.336870, version=8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, architecture=x86_64, maintainer=Guillaume Abrioux , RELEASE=HEAD, io.openshift.tags=base centos centos-stream, name=centos-stream) 2024-04-07T14:11:35.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.11 MB/sec execute 57 sec latency 39970.988 ms 2024-04-07T14:11:35.830 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:35.578+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:35.830 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:35.578+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:35.831 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:35.637+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:35.831 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:35.637+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:35.831 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.546362764 +0000 UTC m=+0.410590022 container init ec448797ff779068fc2fc50ef57022b9286175e8843fbbb7cbbf0407058c0af6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4-deactivate, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, version=8, GIT_CLEAN=True, io.openshift.expose-services=, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, com.redhat.component=centos-stream-container, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, GIT_BRANCH=HEAD, io.buildah.version=1.19.8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, distribution-scope=public, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, RELEASE=HEAD) 2024-04-07T14:11:35.831 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.549590484 +0000 UTC m=+0.413817743 container start ec448797ff779068fc2fc50ef57022b9286175e8843fbbb7cbbf0407058c0af6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, ceph=True, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , release=754, GIT_CLEAN=True, io.openshift.expose-services=, io.buildah.version=1.19.8, version=8, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., vcs-type=git, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, build-date=2022-05-03T08:36:31.336870, name=centos-stream, com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac) 2024-04-07T14:11:35.832 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.621144697 +0000 UTC m=+0.485372144 container attach ec448797ff779068fc2fc50ef57022b9286175e8843fbbb7cbbf0407058c0af6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4-deactivate, io.openshift.tags=base centos centos-stream, distribution-scope=public, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_CLEAN=True, com.redhat.component=centos-stream-container, vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, release=754, RELEASE=HEAD, ceph=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, GIT_BRANCH=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, build-date=2022-05-03T08:36:31.336870, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc.) 2024-04-07T14:11:35.832 INFO:journalctl@ceph.osd.4.smithi145.stdout:Apr 07 14:11:35 smithi145 podman[44258]: 2024-04-07 14:11:35.83153928 +0000 UTC m=+0.695766543 container died ec448797ff779068fc2fc50ef57022b9286175e8843fbbb7cbbf0407058c0af6 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-4-deactivate, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, GIT_BRANCH=HEAD, io.openshift.expose-services=, CEPH_POINT_RELEASE=-17.2.0, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.display-name=CentOS Stream 8, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, version=8, vcs-type=git, distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., ceph=True) 2024-04-07T14:11:36.123 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:35.883+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:36.123 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:35 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:35.883+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:36.539 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.4.service' 2024-04-07T14:11:36.540 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 5.02 MB/sec execute 58 sec latency 40971.032 ms 2024-04-07T14:11:36.622 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:36.622 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-07T14:11:36.622 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-07T14:11:36.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.5 2024-04-07T14:11:36.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:36.607+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:36.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:36.607+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:36.773 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:36.610+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:36.774 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:36.610+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:36.774 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 systemd[1]: Stopping Ceph osd.5 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:36.774 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 bash[44338]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.5" found: no such container 2024-04-07T14:11:37.273 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:36.869+0000 7fbdf35c2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:37.273 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:36.869+0000 7fbdf35c2700 -1 osd.5 70 *** Got signal Terminated *** 2024-04-07T14:11:37.273 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:36.869+0000 7fbdf35c2700 -1 osd.5 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:37.274 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:36.855+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:37.274 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:36 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:36.855+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:37.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.94 MB/sec execute 59 sec latency 41971.074 ms 2024-04-07T14:11:38.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:37.591+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:38.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:37.591+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:38.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:37.831+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:38.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:37.831+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:38.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:37.595+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:38.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:37 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:37.595+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:38.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.86 MB/sec execute 60 sec latency 42971.118 ms 2024-04-07T14:11:39.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:38.629+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:39.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:38.629+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:39.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:38.612+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:39.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:38.612+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:39.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:38.844+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:39.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:38 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:38.844+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:39.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.78 MB/sec execute 61 sec latency 43971.134 ms 2024-04-07T14:11:39.892 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:39.651+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:39.892 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:39.651+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:39.893 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:39.633+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:39.893 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:39.633+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:40.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:39.892+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:40.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:39 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:39.892+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:40.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.70 MB/sec execute 62 sec latency 44971.177 ms 2024-04-07T14:11:41.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:40.860+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:41.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:40.860+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:41.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:40.667+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:41.024 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:40.667+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:41.024 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:40.667+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.454073+0000 front 2024-04-07T14:11:17.447712+0000 (oldest deadline 2024-04-07T14:11:40.347755+0000) 2024-04-07T14:11:41.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:40.682+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:41.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:40.682+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:41.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:40 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:40.682+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:41.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.63 MB/sec execute 63 sec latency 45971.232 ms 2024-04-07T14:11:42.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:41.665+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:10:58.844269+0000 front 2024-04-07T14:10:58.844543+0000 (oldest deadline 2024-04-07T14:11:22.344484+0000) 2024-04-07T14:11:42.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:41.665+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.445538+0000 front 2024-04-07T14:11:06.445472+0000 (oldest deadline 2024-04-07T14:11:31.046246+0000) 2024-04-07T14:11:42.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5[33992]: 2024-04-07T14:11:41.665+0000 7fbdedf59700 -1 osd.5 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.454073+0000 front 2024-04-07T14:11:17.447712+0000 (oldest deadline 2024-04-07T14:11:40.347755+0000) 2024-04-07T14:11:42.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:41.838+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:42.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:41.838+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:42.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:41.649+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:42.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:41.649+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:42.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:41 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:41.649+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:42.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.55 MB/sec execute 64 sec latency 46971.251 ms 2024-04-07T14:11:42.704 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:42 smithi145 podman[44349]: 2024-04-07 14:11:42.407053021 +0000 UTC m=+5.663090687 container died 5ffb10ce07994f40e8c80af467d935e1e7a88d5766c3ac0807260fd921dca58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5, io.buildah.version=1.19.8, version=8, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, name=centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, GIT_BRANCH=HEAD, architecture=x86_64, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.expose-services=, maintainer=Guillaume Abrioux , ceph=True, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, vendor=Red Hat, Inc., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, vcs-type=git) 2024-04-07T14:11:42.704 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:42 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:42.599+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:42.704 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:42 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:42.599+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:42.705 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:42 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:42.599+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:43.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:42 smithi145 podman[44349]: 2024-04-07 14:11:42.887123184 +0000 UTC m=+6.143160919 container cleanup 5ffb10ce07994f40e8c80af467d935e1e7a88d5766c3ac0807260fd921dca58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5, version=8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, architecture=x86_64, name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-07T14:11:43.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:42 smithi145 bash[44349]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5 2024-04-07T14:11:43.023 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:42 smithi145 bash[44378]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.5" found: no such container 2024-04-07T14:11:43.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:42 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:42.806+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:43.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:42 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:42.806+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:43.457 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44366]: 2024-04-07 14:11:43.14677674 +0000 UTC m=+0.737879720 container remove 5ffb10ce07994f40e8c80af467d935e1e7a88d5766c3ac0807260fd921dca58a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, ceph=True, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.display-name=CentOS Stream 8, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.expose-services=, vcs-type=git, vendor=Red Hat, Inc., release=754, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, GIT_BRANCH=HEAD, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/) 2024-04-07T14:11:43.457 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.225764456 +0000 UTC m=+0.018713414 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:43.457 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.347687186 +0000 UTC m=+0.140636177 container create 3c726d3fbd40b467f0f5ffc56aa2aa1ff7debb3f79a333066bd1888909ef1a49 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5-deactivate, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, io.buildah.version=1.19.8, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, release=754, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vendor=Red Hat, Inc., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, version=8, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64) 2024-04-07T14:11:43.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.48 MB/sec execute 65 sec latency 47971.315 ms 2024-04-07T14:11:43.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:43.584+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:43.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:43.584+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:43.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:43.584+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:43.774 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.574948809 +0000 UTC m=+0.367897753 container init 3c726d3fbd40b467f0f5ffc56aa2aa1ff7debb3f79a333066bd1888909ef1a49 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., distribution-scope=public, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, version=8, release=754, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_REPO=https://github.com/ceph/ceph-container.git, io.k8s.display-name=CentOS Stream 8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.openshift.expose-services=, architecture=x86_64, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, GIT_CLEAN=True, RELEASE=HEAD) 2024-04-07T14:11:43.775 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.578183734 +0000 UTC m=+0.371132682 container start 3c726d3fbd40b467f0f5ffc56aa2aa1ff7debb3f79a333066bd1888909ef1a49 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5-deactivate, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, distribution-scope=public, io.openshift.expose-services=, name=centos-stream, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-type=git, CEPH_POINT_RELEASE=-17.2.0, ceph=True, release=754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., io.buildah.version=1.19.8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, com.redhat.component=centos-stream-container, architecture=x86_64) 2024-04-07T14:11:43.775 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.683245256 +0000 UTC m=+0.476194205 container attach 3c726d3fbd40b467f0f5ffc56aa2aa1ff7debb3f79a333066bd1888909ef1a49 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, vcs-type=git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, architecture=x86_64, GIT_BRANCH=HEAD, vendor=Red Hat, Inc., io.buildah.version=1.19.8, RELEASE=HEAD, ceph=True, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, name=centos-stream, com.redhat.component=centos-stream-container, distribution-scope=public, build-date=2022-05-03T08:36:31.336870, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, GIT_CLEAN=True, io.k8s.display-name=CentOS Stream 8, CEPH_POINT_RELEASE=-17.2.0, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, release=754, version=8) 2024-04-07T14:11:44.211 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:43.779+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:44.211 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:43.779+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:44.211 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:43 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:43.779+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:44.212 INFO:journalctl@ceph.osd.5.smithi145.stdout:Apr 07 14:11:43 smithi145 podman[44446]: 2024-04-07 14:11:43.865989933 +0000 UTC m=+0.658938870 container died 3c726d3fbd40b467f0f5ffc56aa2aa1ff7debb3f79a333066bd1888909ef1a49 (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-5-deactivate, GIT_REPO=https://github.com/ceph/ceph-container.git, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, io.k8s.display-name=CentOS Stream 8, architecture=x86_64, vendor=Red Hat, Inc., GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-type=git, distribution-scope=public, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, version=8, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, RELEASE=HEAD, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, name=centos-stream, release=754) 2024-04-07T14:11:44.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.42 MB/sec execute 66 sec latency 48971.374 ms 2024-04-07T14:11:44.623 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.5.service' 2024-04-07T14:11:44.664 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:44.665 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-07T14:11:44.665 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-07T14:11:44.665 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.6 2024-04-07T14:11:44.881 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.739+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:44.882 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.739+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:44.882 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.739+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:44.882 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 systemd[1]: Stopping Ceph osd.6 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:44.882 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 bash[44524]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.6" found: no such container 2024-04-07T14:11:44.883 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:44.555+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:44.883 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:44.555+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:44.883 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:44.555+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:45.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.881+0000 7f29024ed700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:45.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.881+0000 7f29024ed700 -1 osd.6 70 *** Got signal Terminated *** 2024-04-07T14:11:45.273 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:44 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:44.881+0000 7f29024ed700 -1 osd.6 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:45.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.35 MB/sec execute 67 sec latency 49971.417 ms 2024-04-07T14:11:46.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:45.724+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:46.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:45.724+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:46.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:45.724+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:46.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:45.546+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:46.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:45.546+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:46.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:45 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:45.546+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:46.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.29 MB/sec execute 68 sec latency 50971.467 ms 2024-04-07T14:11:47.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:46.742+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:47.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:46.742+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:47.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:46.742+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:47.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:46.551+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:47.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:46.551+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:47.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:46 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:46.551+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:47.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.22 MB/sec execute 69 sec latency 51971.518 ms 2024-04-07T14:11:47.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:47.727+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:47.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:47.727+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:47.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:47.727+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:47.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:47.727+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:23.236003+0000 front 2024-04-07T14:11:23.236037+0000 (oldest deadline 2024-04-07T14:11:47.336080+0000) 2024-04-07T14:11:47.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:47.513+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:47.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:47.513+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:47.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:47 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:47.513+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:48.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.16 MB/sec execute 70 sec latency 52971.561 ms 2024-04-07T14:11:49.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:48.729+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:49.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:48.729+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:48.729+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:48.729+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:23.236003+0000 front 2024-04-07T14:11:23.236037+0000 (oldest deadline 2024-04-07T14:11:47.336080+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:48.554+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:48.554+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:48.554+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:49.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:48 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:48.554+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:49.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.10 MB/sec execute 71 sec latency 53971.613 ms 2024-04-07T14:11:50.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:49.689+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.632135+0000 front 2024-04-07T14:11:01.632035+0000 (oldest deadline 2024-04-07T14:11:25.732107+0000) 2024-04-07T14:11:50.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:49.689+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:06.236988+0000 front 2024-04-07T14:11:06.233359+0000 (oldest deadline 2024-04-07T14:11:31.533507+0000) 2024-04-07T14:11:50.023 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:49.689+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.935237+0000 front 2024-04-07T14:11:17.938976+0000 (oldest deadline 2024-04-07T14:11:43.235410+0000) 2024-04-07T14:11:50.024 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6[36971]: 2024-04-07T14:11:49.689+0000 7f28fd685700 -1 osd.6 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:23.236003+0000 front 2024-04-07T14:11:23.236037+0000 (oldest deadline 2024-04-07T14:11:47.336080+0000) 2024-04-07T14:11:50.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:49.532+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:50.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:49.532+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:50.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:49.532+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:50.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:49 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:49.532+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:50.514 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:50 smithi145 podman[44536]: 2024-04-07 14:11:50.256559338 +0000 UTC m=+5.481722992 container died 5e283a14fbad9df85ba19782abf38f547500160ab56942b74786603489eb45fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6, io.openshift.tags=base centos centos-stream, vendor=Red Hat, Inc., io.buildah.version=1.19.8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.openshift.expose-services=, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, ceph=True, CEPH_POINT_RELEASE=-17.2.0, distribution-scope=public, GIT_CLEAN=True, name=centos-stream, vcs-type=git, io.k8s.display-name=CentOS Stream 8, GIT_REPO=https://github.com/ceph/ceph-container.git, release=754, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb) 2024-04-07T14:11:50.515 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:50 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:50.494+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:50.515 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:50 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:50.494+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:50.515 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:50 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:50.494+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:50.515 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:50 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:50.494+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:50.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 4.05 MB/sec execute 72 sec latency 54971.666 ms 2024-04-07T14:11:50.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:50 smithi145 podman[44536]: 2024-04-07 14:11:50.614728786 +0000 UTC m=+5.839892435 container cleanup 5e283a14fbad9df85ba19782abf38f547500160ab56942b74786603489eb45fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6, release=754, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, name=centos-stream, version=8, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, maintainer=Guillaume Abrioux , build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, CEPH_POINT_RELEASE=-17.2.0, GIT_REPO=https://github.com/ceph/ceph-container.git, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, distribution-scope=public, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.expose-services=, vcs-type=git, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_CLEAN=True) 2024-04-07T14:11:50.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:50 smithi145 bash[44536]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6 2024-04-07T14:11:50.773 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:50 smithi145 bash[44565]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.6" found: no such container 2024-04-07T14:11:51.197 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:50 smithi145 podman[44551]: 2024-04-07 14:11:50.907764617 +0000 UTC m=+0.655525416 container remove 5e283a14fbad9df85ba19782abf38f547500160ab56942b74786603489eb45fe (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, distribution-scope=public, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, name=centos-stream, io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.component=centos-stream-container, GIT_BRANCH=HEAD, version=8, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_CLEAN=True, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, release=754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vendor=Red Hat, Inc., ceph=True) 2024-04-07T14:11:51.197 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.005774084 +0000 UTC m=+0.018176426 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:51.197 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.116893546 +0000 UTC m=+0.129295869 container create a0debe7689ee2e7095b1755b5b6a339f46f12235247dc5aedcd2f117934a739e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6-deactivate, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, distribution-scope=public, RELEASE=HEAD, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, version=8, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, CEPH_POINT_RELEASE=-17.2.0, vcs-type=git, io.buildah.version=1.19.8, GIT_BRANCH=HEAD, build-date=2022-05-03T08:36:31.336870, io.openshift.tags=base centos centos-stream, release=754, GIT_CLEAN=True, io.openshift.expose-services=, io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , architecture=x86_64, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, vendor=Red Hat, Inc.) 2024-04-07T14:11:51.523 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.327468852 +0000 UTC m=+0.339871176 container init a0debe7689ee2e7095b1755b5b6a339f46f12235247dc5aedcd2f117934a739e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6-deactivate, maintainer=Guillaume Abrioux , io.openshift.expose-services=, architecture=x86_64, ceph=True, release=754, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_BRANCH=HEAD, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, distribution-scope=public, RELEASE=HEAD, vendor=Red Hat, Inc., io.k8s.display-name=CentOS Stream 8, com.redhat.component=centos-stream-container, name=centos-stream) 2024-04-07T14:11:51.523 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.330575543 +0000 UTC m=+0.342977876 container start a0debe7689ee2e7095b1755b5b6a339f46f12235247dc5aedcd2f117934a739e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, architecture=x86_64, io.k8s.display-name=CentOS Stream 8, vcs-type=git, RELEASE=HEAD, ceph=True, version=8, GIT_BRANCH=HEAD, GIT_REPO=https://github.com/ceph/ceph-container.git, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, maintainer=Guillaume Abrioux , vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, CEPH_POINT_RELEASE=-17.2.0, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., build-date=2022-05-03T08:36:31.336870, com.redhat.component=centos-stream-container, name=centos-stream, io.openshift.expose-services=, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_CLEAN=True, distribution-scope=public) 2024-04-07T14:11:51.523 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.410741066 +0000 UTC m=+0.423143407 container attach a0debe7689ee2e7095b1755b5b6a339f46f12235247dc5aedcd2f117934a739e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6-deactivate, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, distribution-scope=public, maintainer=Guillaume Abrioux , com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, release=754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, version=8, architecture=x86_64, build-date=2022-05-03T08:36:31.336870, name=centos-stream, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, vcs-type=git, GIT_CLEAN=True, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T14:11:51.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.99 MB/sec execute 73 sec latency 55971.715 ms 2024-04-07T14:11:51.913 INFO:journalctl@ceph.osd.6.smithi145.stdout:Apr 07 14:11:51 smithi145 podman[44633]: 2024-04-07 14:11:51.620038165 +0000 UTC m=+0.632440499 container died a0debe7689ee2e7095b1755b5b6a339f46f12235247dc5aedcd2f117934a739e (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-6-deactivate, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, GIT_CLEAN=True, RELEASE=HEAD, architecture=x86_64, vcs-type=git, com.redhat.component=centos-stream-container, GIT_REPO=https://github.com/ceph/ceph-container.git, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, maintainer=Guillaume Abrioux , url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.openshift.tags=base centos centos-stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, CEPH_POINT_RELEASE=-17.2.0, version=8, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, io.k8s.display-name=CentOS Stream 8, ceph=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, name=centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com) 2024-04-07T14:11:51.913 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:51.526+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:51.913 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:51.526+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:51.913 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:51.526+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:51.913 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:51 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:51.526+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:52.206 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.6.service' 2024-04-07T14:11:52.247 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:11:52.247 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-07T14:11:52.247 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-07T14:11:52.247 DEBUG:teuthology.orchestra.run.smithi145:> sudo systemctl stop ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.7 2024-04-07T14:11:52.474 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 systemd[1]: Stopping Ceph osd.7 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4... 2024-04-07T14:11:52.475 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 bash[44711]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.7" found: no such container 2024-04-07T14:11:52.475 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.474+0000 7f27043ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-07T14:11:52.475 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.474+0000 7f27043ba700 -1 osd.7 70 *** Got signal Terminated *** 2024-04-07T14:11:52.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.94 MB/sec execute 74 sec latency 56971.764 ms 2024-04-07T14:11:52.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.474+0000 7f27043ba700 -1 osd.7 70 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-07T14:11:52.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.539+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:52.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.539+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:52.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.539+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:52.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:52 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:52.539+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:53.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.89 MB/sec execute 75 sec latency 57971.830 ms 2024-04-07T14:11:53.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:53 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:53.490+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:53.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:53 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:53.490+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:53.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:53 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:53.490+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:53.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:53 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:53.490+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:54.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.83 MB/sec execute 76 sec latency 58971.881 ms 2024-04-07T14:11:54.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:54 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:54.481+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:54.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:54 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:54.481+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:54.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:54 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:54.481+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:54.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:54 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:54.481+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:55.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.79 MB/sec execute 77 sec latency 59971.920 ms 2024-04-07T14:11:55.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:55.515+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:55.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:55.515+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:55.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:55.515+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:55.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:55.515+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:55.774 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:55 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:55.515+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.145:6804 osd.4 since back 2024-04-07T14:11:33.704358+0000 front 2024-04-07T14:11:33.704377+0000 (oldest deadline 2024-04-07T14:11:55.404422+0000) 2024-04-07T14:11:56.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.74 MB/sec execute 78 sec latency 60971.944 ms 2024-04-07T14:11:57.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:56 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:56.548+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6806 osd.0 since back 2024-04-07T14:11:01.604951+0000 front 2024-04-07T14:11:01.600015+0000 (oldest deadline 2024-04-07T14:11:25.099986+0000) 2024-04-07T14:11:57.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:56 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:56.548+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6814 osd.1 since back 2024-04-07T14:11:07.400747+0000 front 2024-04-07T14:11:07.400813+0000 (oldest deadline 2024-04-07T14:11:32.700855+0000) 2024-04-07T14:11:57.023 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:56 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:56.548+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6822 osd.2 since back 2024-04-07T14:11:17.302241+0000 front 2024-04-07T14:11:17.302223+0000 (oldest deadline 2024-04-07T14:11:40.202448+0000) 2024-04-07T14:11:57.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:56 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:56.548+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.111:6830 osd.3 since back 2024-04-07T14:11:24.303460+0000 front 2024-04-07T14:11:24.306825+0000 (oldest deadline 2024-04-07T14:11:47.803482+0000) 2024-04-07T14:11:57.024 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:56 smithi145 ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7[40042]: 2024-04-07T14:11:56.548+0000 7f2701d57700 -1 osd.7 70 heartbeat_check: no reply from 172.21.15.145:6804 osd.4 since back 2024-04-07T14:11:33.704358+0000 front 2024-04-07T14:11:33.704377+0000 (oldest deadline 2024-04-07T14:11:55.404422+0000) 2024-04-07T14:11:57.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.69 MB/sec execute 79 sec latency 61971.984 ms 2024-04-07T14:11:58.383 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 podman[44725]: 2024-04-07 14:11:58.089937371 +0000 UTC m=+5.722690253 container died d5503daa3f4778eafceefb0c8c8569e15aa5890879f39bf36f11b14a4712e28a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7, GIT_CLEAN=True, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, version=8, release=754, GIT_BRANCH=HEAD, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., distribution-scope=public, name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, vendor=Red Hat, Inc., io.buildah.version=1.19.8, ceph=True, build-date=2022-05-03T08:36:31.336870, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, io.k8s.display-name=CentOS Stream 8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, com.redhat.component=centos-stream-container, maintainer=Guillaume Abrioux , vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, io.openshift.tags=base centos centos-stream, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=) 2024-04-07T14:11:58.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.64 MB/sec execute 80 sec latency 62972.035 ms 2024-04-07T14:11:58.741 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 podman[44725]: 2024-04-07 14:11:58.491494065 +0000 UTC m=+6.124246867 container cleanup d5503daa3f4778eafceefb0c8c8569e15aa5890879f39bf36f11b14a4712e28a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7, build-date=2022-05-03T08:36:31.336870, io.buildah.version=1.19.8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, com.redhat.component=centos-stream-container, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, architecture=x86_64, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vendor=Red Hat, Inc., GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, io.openshift.expose-services=, RELEASE=HEAD, ceph=True, version=8, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, release=754, CEPH_POINT_RELEASE=-17.2.0, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, maintainer=Guillaume Abrioux , name=centos-stream, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_CLEAN=True, GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public) 2024-04-07T14:11:58.742 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 bash[44725]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7 2024-04-07T14:11:58.742 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 bash[44756]: Error: no container with name or ID "ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd.7" found: no such container 2024-04-07T14:11:59.001 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 podman[44744]: 2024-04-07 14:11:58.742667529 +0000 UTC m=+0.647140722 container remove d5503daa3f4778eafceefb0c8c8569e15aa5890879f39bf36f11b14a4712e28a (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., RELEASE=HEAD, io.buildah.version=1.19.8, io.openshift.expose-services=, build-date=2022-05-03T08:36:31.336870, version=8, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, architecture=x86_64, distribution-scope=public, CEPH_POINT_RELEASE=-17.2.0, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_BRANCH=HEAD, io.openshift.tags=base centos centos-stream, vcs-type=git, vendor=Red Hat, Inc., name=centos-stream, com.redhat.component=centos-stream-container, release=754, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, ceph=True, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_REPO=https://github.com/ceph/ceph-container.git) 2024-04-07T14:11:59.001 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:58 smithi145 podman[44822]: 2024-04-07 14:11:58.886773121 +0000 UTC m=+0.018626371 image pull e1d6a67b021eb077ee22bf650f1a9fb1980a2cf5c36bdb9cba9eac6de8f702d9 quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a 2024-04-07T14:11:59.001 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:59 smithi145 podman[44822]: 2024-04-07 14:11:59.001896512 +0000 UTC m=+0.133749750 container create 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., release=754, build-date=2022-05-03T08:36:31.336870, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, architecture=x86_64, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, com.redhat.component=centos-stream-container, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.openshift.tags=base centos centos-stream, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.buildah.version=1.19.8, version=8, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, ceph=True, io.k8s.display-name=CentOS Stream 8, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, CEPH_POINT_RELEASE=-17.2.0, RELEASE=HEAD, io.openshift.expose-services=, distribution-scope=public) 2024-04-07T14:11:59.273 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:59 smithi145 podman[44822]: 2024-04-07 14:11:59.221401528 +0000 UTC m=+0.353254762 container init 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, GIT_BRANCH=HEAD, vcs-type=git, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8, io.openshift.expose-services=, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, io.openshift.tags=base centos centos-stream, RELEASE=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, version=8, GIT_CLEAN=True, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, ceph=True, release=754, io.k8s.display-name=CentOS Stream 8, maintainer=Guillaume Abrioux , GIT_REPO=https://github.com/ceph/ceph-container.git, distribution-scope=public, name=centos-stream, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, architecture=x86_64, vendor=Red Hat, Inc., url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754) 2024-04-07T14:11:59.273 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:59 smithi145 podman[44822]: 2024-04-07 14:11:59.224359405 +0000 UTC m=+0.356212639 container start 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, io.k8s.display-name=CentOS Stream 8, distribution-scope=public, vendor=Red Hat, Inc., CEPH_POINT_RELEASE=-17.2.0, version=8, ceph=True, architecture=x86_64, vcs-type=git, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, io.openshift.tags=base centos centos-stream, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, com.redhat.component=centos-stream-container, name=centos-stream, release=754, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, io.openshift.expose-services=, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , com.redhat.license_terms=https://centos.org/legal/licensing-policy/, build-date=2022-05-03T08:36:31.336870, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, io.buildah.version=1.19.8) 2024-04-07T14:11:59.541 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.60 MB/sec execute 81 sec latency 63972.112 ms 2024-04-07T14:11:59.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:59 smithi145 podman[44822]: 2024-04-07 14:11:59.313003378 +0000 UTC m=+0.444856622 container attach 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, io.openshift.expose-services=, io.openshift.tags=base centos centos-stream, GIT_CLEAN=True, GIT_BRANCH=HEAD, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, name=centos-stream, GIT_REPO=https://github.com/ceph/ceph-container.git, vendor=Red Hat, Inc., com.redhat.license_terms=https://centos.org/legal/licensing-policy/, RELEASE=HEAD, vcs-type=git, architecture=x86_64, CEPH_POINT_RELEASE=-17.2.0, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, com.redhat.component=centos-stream-container, version=8, release=754, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, distribution-scope=public, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, build-date=2022-05-03T08:36:31.336870, io.k8s.display-name=CentOS Stream 8, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., maintainer=Guillaume Abrioux , io.buildah.version=1.19.8) 2024-04-07T14:11:59.773 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:11:59 smithi145 podman[44822]: 2024-04-07 14:11:59.501447313 +0000 UTC m=+0.633300550 container died 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, ceph=True, version=8, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, architecture=x86_64, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., name=centos-stream, CEPH_POINT_RELEASE=-17.2.0, io.k8s.display-name=CentOS Stream 8, vcs-type=git, com.redhat.component=centos-stream-container, vendor=Red Hat, Inc., io.openshift.tags=base centos centos-stream, io.buildah.version=1.19.8, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, maintainer=Guillaume Abrioux , io.openshift.expose-services=, GIT_REPO=https://github.com/ceph/ceph-container.git, RELEASE=HEAD, build-date=2022-05-03T08:36:31.336870, GIT_CLEAN=True, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, GIT_BRANCH=HEAD, distribution-scope=public, release=754) 2024-04-07T14:12:00.053 DEBUG:teuthology.orchestra.run.smithi145:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.7.service' 2024-04-07T14:12:00.078 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:12:00 smithi145 podman[44848]: 2024-04-07 14:12:00.035064986 +0000 UTC m=+0.528291420 container remove 3d8cf0e3a6e0279507db2c314657ceedf925f57cca13bc8dab7786d526186b3d (image=quay.io/ceph/ceph@sha256:12a0a4f43413fd97a14a3d47a3451b2d2df50020835bb93db666209f3f77617a, name=ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4-osd-7-deactivate, ceph=True, GIT_COMMIT=b613db0f44179c0940781c1c7fe04e1acb7093ac, build-date=2022-05-03T08:36:31.336870, io.openshift.expose-services=, io.k8s.description=The Universal Base Image is designed and engineered to be the base layer for all of your containerized applications, middleware and utilities. This base image is freely redistributable, but Red Hat only supports Red Hat technologies through subscriptions for Red Hat products. This image is maintained by Red Hat and updated regularly., vcs-type=git, com.redhat.build-host=cpt-1002.osbs.prod.upshift.rdu2.redhat.com, CEPH_POINT_RELEASE=-17.2.0, release=754, architecture=x86_64, summary=Provides a CentOS Stream container based on the Red Hat Universal Base Image, GIT_REPO=https://github.com/ceph/ceph-container.git, version=8, io.k8s.display-name=CentOS Stream 8, RELEASE=HEAD, com.redhat.license_terms=https://centos.org/legal/licensing-policy/, distribution-scope=public, GIT_CLEAN=True, maintainer=Guillaume Abrioux , com.redhat.component=centos-stream-container, url=https://access.redhat.com/containers/#/registry.access.redhat.com/ubi8/images/8.6-754, GIT_BRANCH=HEAD, description=CentOS Stream is a continuously delivered distro that tracks just ahead of Red Hat Enterprise Linux development. This image takes the Red Hat UBI and layers on content from CentOS Stream, name=centos-stream, io.openshift.tags=base centos centos-stream, vcs-ref=f1ee6e37554363ec55e0035aba1a693d3627fdeb, vendor=Red Hat, Inc., io.buildah.version=1.19.8) 2024-04-07T14:12:00.079 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:12:00 smithi145 systemd[1]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.7.service: Deactivated successfully. 2024-04-07T14:12:00.079 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:12:00 smithi145 systemd[1]: Stopped Ceph osd.7 for 2bfa0756-f4e7-11ee-b649-cb9ed24678a4. 2024-04-07T14:12:00.079 INFO:journalctl@ceph.osd.7.smithi145.stdout:Apr 07 14:12:00 smithi145 systemd[1]: ceph-2bfa0756-f4e7-11ee-b649-cb9ed24678a4@osd.7.service: Consumed 1min 26.344s CPU time. 2024-04-07T14:12:00.092 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:12:00.093 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-07T14:12:00.093 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-07T14:12:00.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.55 MB/sec execute 82 sec latency 64972.163 ms 2024-04-07T14:12:01.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.51 MB/sec execute 83 sec latency 65972.214 ms 2024-04-07T14:12:02.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.47 MB/sec execute 84 sec latency 66972.239 ms 2024-04-07T14:12:03.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.43 MB/sec execute 85 sec latency 67972.289 ms 2024-04-07T14:12:04.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.39 MB/sec execute 86 sec latency 68972.335 ms 2024-04-07T14:12:05.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.35 MB/sec execute 87 sec latency 69972.614 ms 2024-04-07T14:12:06.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.31 MB/sec execute 88 sec latency 70972.641 ms 2024-04-07T14:12:07.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.27 MB/sec execute 89 sec latency 71972.667 ms 2024-04-07T14:12:08.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.24 MB/sec execute 90 sec latency 72972.719 ms 2024-04-07T14:12:09.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.20 MB/sec execute 91 sec latency 73972.753 ms 2024-04-07T14:12:10.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.17 MB/sec execute 92 sec latency 74972.782 ms 2024-04-07T14:12:11.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.13 MB/sec execute 93 sec latency 75972.832 ms 2024-04-07T14:12:12.546 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.10 MB/sec execute 94 sec latency 76972.855 ms 2024-04-07T14:12:13.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.07 MB/sec execute 95 sec latency 77972.884 ms 2024-04-07T14:12:14.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.04 MB/sec execute 96 sec latency 78972.940 ms 2024-04-07T14:12:15.542 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 3.00 MB/sec execute 97 sec latency 79972.980 ms 2024-04-07T14:12:16.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.97 MB/sec execute 98 sec latency 80973.018 ms 2024-04-07T14:12:17.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.94 MB/sec execute 99 sec latency 81973.041 ms 2024-04-07T14:12:18.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.91 MB/sec execute 100 sec latency 82973.082 ms 2024-04-07T14:12:19.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.89 MB/sec execute 101 sec latency 83973.130 ms 2024-04-07T14:12:20.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.86 MB/sec execute 102 sec latency 84973.161 ms 2024-04-07T14:12:21.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.83 MB/sec execute 103 sec latency 85973.208 ms 2024-04-07T14:12:22.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.80 MB/sec execute 104 sec latency 86973.237 ms 2024-04-07T14:12:23.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.78 MB/sec execute 105 sec latency 87973.281 ms 2024-04-07T14:12:24.361 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 --force --keep-logs 2024-04-07T14:12:24.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.75 MB/sec execute 106 sec latency 88973.333 ms 2024-04-07T14:12:25.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.72 MB/sec execute 107 sec latency 89973.373 ms 2024-04-07T14:12:26.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.70 MB/sec execute 108 sec latency 90973.419 ms 2024-04-07T14:12:27.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.67 MB/sec execute 109 sec latency 91973.450 ms 2024-04-07T14:12:28.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.65 MB/sec execute 110 sec latency 92973.514 ms 2024-04-07T14:12:29.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.63 MB/sec execute 111 sec latency 93973.589 ms 2024-04-07T14:12:30.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.60 MB/sec execute 112 sec latency 94973.689 ms 2024-04-07T14:12:31.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.58 MB/sec execute 113 sec latency 95973.785 ms 2024-04-07T14:12:32.543 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.56 MB/sec execute 114 sec latency 96973.844 ms 2024-04-07T14:12:33.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.53 MB/sec execute 115 sec latency 97973.999 ms 2024-04-07T14:12:34.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.51 MB/sec execute 116 sec latency 98974.100 ms 2024-04-07T14:12:35.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.49 MB/sec execute 117 sec latency 99974.175 ms 2024-04-07T14:12:36.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.47 MB/sec execute 118 sec latency 100974.263 ms 2024-04-07T14:12:37.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.45 MB/sec execute 119 sec latency 101974.375 ms 2024-04-07T14:12:38.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.43 MB/sec execute 120 sec latency 102974.431 ms 2024-04-07T14:12:39.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.41 MB/sec execute 121 sec latency 103974.500 ms 2024-04-07T14:12:40.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.39 MB/sec execute 122 sec latency 104974.567 ms 2024-04-07T14:12:41.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.37 MB/sec execute 123 sec latency 105974.649 ms 2024-04-07T14:12:42.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.35 MB/sec execute 124 sec latency 106974.747 ms 2024-04-07T14:12:43.337 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:12:43.371 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-07T14:12:43.400 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-07T14:12:43.402 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi111/crash 2024-04-07T14:12:43.403 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash -- . 2024-04-07T14:12:43.452 INFO:teuthology.orchestra.run.smithi111.stderr:tar: /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-07T14:12:43.452 INFO:teuthology.orchestra.run.smithi111.stderr:tar: Error is not recoverable: exiting now 2024-04-07T14:12:43.454 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi145/crash 2024-04-07T14:12:43.455 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash -- . 2024-04-07T14:12:43.500 INFO:teuthology.orchestra.run.smithi145.stderr:tar: /var/lib/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/crash: Cannot open: No such file or directory 2024-04-07T14:12:43.500 INFO:teuthology.orchestra.run.smithi145.stderr:tar: Error is not recoverable: exiting now 2024-04-07T14:12:43.501 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-07T14:12:43.502 DEBUG:teuthology.orchestra.run.smithi111:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2024-04-07T14:12:43.532 INFO:teuthology.orchestra.run.smithi111.stdout:2024-04-07T14:05:56.351278+0000 mon.smithi111 (mon.0) 514 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-04-07T14:12:43.533 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-04-07T14:12:43.533 INFO:tasks.cephadm:Compressing logs... 2024-04-07T14:12:43.533 DEBUG:teuthology.orchestra.run.smithi111:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-07T14:12:43.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.33 MB/sec execute 125 sec latency 107974.786 ms 2024-04-07T14:12:43.546 DEBUG:teuthology.orchestra.run.smithi145:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-07T14:12:43.581 INFO:teuthology.orchestra.run.smithi111.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-07T14:12:43.581 INFO:teuthology.orchestra.run.smithi111.stderr:: No such file or directory 2024-04-07T14:12:43.581 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi111.pvxsdw.log 2024-04-07T14:12:43.582 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-07T14:12:43.582 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi111.pvxsdw.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi111.rtrlso.log 2024-04-07T14:12:43.582 INFO:teuthology.orchestra.run.smithi111.stderr: 80.0% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi111.pvxsdw.log.gz 2024-04-07T14:12:43.582 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.1.log 2024-04-07T14:12:43.583 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.3.log 2024-04-07T14:12:43.583 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi111.rtrlso.log: /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.0.log 2024-04-07T14:12:43.583 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi111.log 2024-04-07T14:12:43.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-07T14:12:43.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi111.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log 2024-04-07T14:12:43.584 INFO:teuthology.orchestra.run.smithi111.stderr: 90.8% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-07T14:12:43.584 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.2.log 2024-04-07T14:12:43.585 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log: 83.0% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-07T14:12:43.585 INFO:teuthology.orchestra.run.smithi111.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-07T14:12:43.585 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.2.log: 86.2% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-07T14:12:43.588 INFO:teuthology.orchestra.run.smithi111.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log: 91.1% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-07T14:12:43.588 INFO:teuthology.orchestra.run.smithi111.stderr: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-07T14:12:43.593 INFO:teuthology.orchestra.run.smithi145.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-07T14:12:43.593 INFO:teuthology.orchestra.run.smithi145.stderr:: No such file or directory 2024-04-07T14:12:43.593 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.6.log 2024-04-07T14:12:43.593 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi145.qiunpb.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi145.qiunpb.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi145.fczrwb.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.4.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log: /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi145.fczrwb.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi145.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr: 60.4% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mds.foofs.smithi145.fczrwb.log.gz 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.7.log 2024-04-07T14:12:43.594 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi145.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.5.log 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.5.log: 91.0% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.audit.log.gz 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log: 82.3% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.cephadm.log.gz 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:gzip -5 --verbose -- /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log: 86.1% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph.log.gz 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr:/var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log: 90.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr: 91.1% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-volume.log.gz 2024-04-07T14:12:43.595 INFO:teuthology.orchestra.run.smithi145.stderr: 91.6% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi145.qiunpb.log.gz 2024-04-07T14:12:43.677 INFO:teuthology.orchestra.run.smithi111.stderr: 88.4% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mgr.smithi111.rtrlso.log.gz 2024-04-07T14:12:43.729 INFO:teuthology.orchestra.run.smithi145.stderr: 92.6% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi145.log.gz 2024-04-07T14:12:44.013 INFO:teuthology.orchestra.run.smithi111.stderr: 92.1% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-mon.smithi111.log.gz 2024-04-07T14:12:44.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.31 MB/sec execute 126 sec latency 108974.835 ms 2024-04-07T14:12:45.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.29 MB/sec execute 127 sec latency 109974.899 ms 2024-04-07T14:12:46.364 INFO:teuthology.orchestra.run.smithi111.stderr: 94.6% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.2.log.gz 2024-04-07T14:12:46.544 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.28 MB/sec execute 128 sec latency 110974.938 ms 2024-04-07T14:12:47.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.26 MB/sec execute 129 sec latency 111974.977 ms 2024-04-07T14:12:47.774 INFO:teuthology.orchestra.run.smithi145.stderr: 94.3% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.6.log.gz 2024-04-07T14:12:47.873 INFO:teuthology.orchestra.run.smithi145.stderr: 94.1% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.4.log.gz 2024-04-07T14:12:48.326 INFO:teuthology.orchestra.run.smithi111.stderr: 94.2% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.0.log.gz 2024-04-07T14:12:48.507 INFO:teuthology.orchestra.run.smithi111.stderr: 94.4% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.1.log.gz 2024-04-07T14:12:48.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.24 MB/sec execute 130 sec latency 112975.028 ms 2024-04-07T14:12:48.585 INFO:teuthology.orchestra.run.smithi111.stderr: 94.2% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.3.log.gz 2024-04-07T14:12:48.587 INFO:teuthology.orchestra.run.smithi111.stderr: 2024-04-07T14:12:48.587 INFO:teuthology.orchestra.run.smithi111.stderr:real 0m5.028s 2024-04-07T14:12:48.587 INFO:teuthology.orchestra.run.smithi111.stderr:user 0m17.371s 2024-04-07T14:12:48.587 INFO:teuthology.orchestra.run.smithi111.stderr:sys 0m0.683s 2024-04-07T14:12:49.177 INFO:teuthology.orchestra.run.smithi145.stderr: 94.3% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.5.log.gz 2024-04-07T14:12:49.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.22 MB/sec execute 131 sec latency 113975.149 ms 2024-04-07T14:12:50.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.21 MB/sec execute 132 sec latency 114975.262 ms 2024-04-07T14:12:50.905 INFO:teuthology.orchestra.run.smithi145.stderr: 94.2% -- replaced with /var/log/ceph/2bfa0756-f4e7-11ee-b649-cb9ed24678a4/ceph-osd.7.log.gz 2024-04-07T14:12:50.907 INFO:teuthology.orchestra.run.smithi145.stderr: 2024-04-07T14:12:50.907 INFO:teuthology.orchestra.run.smithi145.stderr:real 0m7.347s 2024-04-07T14:12:50.908 INFO:teuthology.orchestra.run.smithi145.stderr:user 0m20.783s 2024-04-07T14:12:50.908 INFO:teuthology.orchestra.run.smithi145.stderr:sys 0m0.824s 2024-04-07T14:12:50.908 INFO:tasks.cephadm:Archiving logs... 2024-04-07T14:12:50.908 DEBUG:teuthology.misc:Transferring archived files from smithi111:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi111/log 2024-04-07T14:12:50.909 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-07T14:12:51.571 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.19 MB/sec execute 133 sec latency 115975.301 ms 2024-04-07T14:12:52.565 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.17 MB/sec execute 134 sec latency 116975.348 ms 2024-04-07T14:12:53.446 DEBUG:teuthology.misc:Transferring archived files from smithi145:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi145/log 2024-04-07T14:12:53.447 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-07T14:12:53.555 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.16 MB/sec execute 135 sec latency 117975.425 ms 2024-04-07T14:12:54.560 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.14 MB/sec execute 136 sec latency 118975.525 ms 2024-04-07T14:12:55.549 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.13 MB/sec execute 137 sec latency 119975.622 ms 2024-04-07T14:12:56.407 INFO:tasks.cephadm:Removing cluster... 2024-04-07T14:12:56.407 DEBUG:teuthology.orchestra.run.smithi111:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 --force 2024-04-07T14:12:56.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.11 MB/sec execute 138 sec latency 120975.686 ms 2024-04-07T14:12:56.703 DEBUG:teuthology.orchestra.run.smithi145:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 --force 2024-04-07T14:12:57.010 INFO:tasks.cephadm:Removing cephadm ... 2024-04-07T14:12:57.010 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-07T14:12:57.025 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-07T14:12:57.039 INFO:tasks.cephadm:Teardown complete 2024-04-07T14:12:57.040 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-07T14:12:57.051 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-07T14:12:57.051 DEBUG:teuthology.orchestra.run.smithi111:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T14:12:57.068 DEBUG:teuthology.orchestra.run.smithi145:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-07T14:12:57.079 INFO:teuthology.orchestra.run.smithi111.stderr:bash: line 1: ntpq: command not found 2024-04-07T14:12:57.083 INFO:teuthology.orchestra.run.smithi111.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T14:12:57.083 INFO:teuthology.orchestra.run.smithi111.stdout:=============================================================================== 2024-04-07T14:12:57.083 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -694us[ -694us] +/- 77ms 2024-04-07T14:12:57.083 INFO:teuthology.orchestra.run.smithi111.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -38us[ -38us] +/- 63ms 2024-04-07T14:12:57.084 INFO:teuthology.orchestra.run.smithi111.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 38 +706us[ +706us] +/- 81ms 2024-04-07T14:12:57.084 INFO:teuthology.orchestra.run.smithi111.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:12:57.093 INFO:teuthology.orchestra.run.smithi145.stderr:bash: line 1: ntpq: command not found 2024-04-07T14:12:57.096 INFO:teuthology.orchestra.run.smithi145.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-07T14:12:57.097 INFO:teuthology.orchestra.run.smithi145.stdout:=============================================================================== 2024-04-07T14:12:57.097 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 37 -715us[ -715us] +/- 78ms 2024-04-07T14:12:57.097 INFO:teuthology.orchestra.run.smithi145.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 -45us[ -57us] +/- 63ms 2024-04-07T14:12:57.097 INFO:teuthology.orchestra.run.smithi145.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 +699us[ +687us] +/- 81ms 2024-04-07T14:12:57.097 INFO:teuthology.orchestra.run.smithi145.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-04-07T14:12:57.098 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-07T14:12:57.107 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-07T14:12:57.108 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-07T14:12:57.118 DEBUG:teuthology.orchestra.run.smithi111:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-07T14:12:57.198 DEBUG:teuthology.orchestra.run.smithi145:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-07T14:12:57.267 DEBUG:teuthology.orchestra.run.smithi111:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-07T14:12:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T14:12:57.298 DEBUG:teuthology.orchestra.run.smithi145:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-07T14:12:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T14:12:57.328 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-07T14:12:57.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-07T14:12:57.348 INFO:teuthology.task.internal:Duration was 1015.031142 seconds 2024-04-07T14:12:57.349 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-07T14:12:57.447 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-07T14:12:57.448 DEBUG:teuthology.orchestra.run.smithi111:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-07T14:12:57.450 DEBUG:teuthology.orchestra.run.smithi145:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-07T14:12:57.491 INFO:teuthology.orchestra.run.smithi111.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T14:12:57.493 INFO:teuthology.orchestra.run.smithi145.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-07T14:12:57.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.10 MB/sec execute 139 sec latency 121975.783 ms 2024-04-07T14:12:57.774 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-07T14:12:57.774 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi111.front.sepia.ceph.com 2024-04-07T14:12:57.775 DEBUG:teuthology.orchestra.run.smithi111:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-07T14:12:57.835 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi145.front.sepia.ceph.com 2024-04-07T14:12:57.835 DEBUG:teuthology.orchestra.run.smithi145:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-07T14:12:57.879 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-07T14:12:57.880 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-07T14:12:57.882 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-07T14:12:57.948 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-07T14:12:57.948 DEBUG:teuthology.orchestra.run.smithi111:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-07T14:12:57.951 DEBUG:teuthology.orchestra.run.smithi145:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-07T14:12:58.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.08 MB/sec execute 140 sec latency 122975.821 ms 2024-04-07T14:12:59.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.07 MB/sec execute 141 sec latency 123975.846 ms 2024-04-07T14:13:00.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.05 MB/sec execute 142 sec latency 124975.891 ms 2024-04-07T14:13:01.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.04 MB/sec execute 143 sec latency 125975.917 ms 2024-04-07T14:13:02.545 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.02 MB/sec execute 144 sec latency 126975.941 ms 2024-04-07T14:13:03.116 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-07T14:13:03.126 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-07T14:13:03.127 DEBUG:teuthology.orchestra.run.smithi111:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-07T14:13:03.153 DEBUG:teuthology.orchestra.run.smithi145:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-07T14:13:03.186 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-07T14:13:03.198 DEBUG:teuthology.orchestra.run.smithi111:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-07T14:13:03.201 DEBUG:teuthology.orchestra.run.smithi145:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-07T14:13:03.223 INFO:teuthology.orchestra.run.smithi111.stdout:kernel.core_pattern = core 2024-04-07T14:13:03.254 INFO:teuthology.orchestra.run.smithi145.stdout:kernel.core_pattern = core 2024-04-07T14:13:03.301 DEBUG:teuthology.orchestra.run.smithi111:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-07T14:13:03.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T14:13:03.315 DEBUG:teuthology.orchestra.run.smithi145:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-07T14:13:03.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-07T14:13:03.357 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-07T14:13:03.370 INFO:teuthology.task.internal:Transferring archived files... 2024-04-07T14:13:03.371 DEBUG:teuthology.misc:Transferring archived files from smithi111:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi111 2024-04-07T14:13:03.371 DEBUG:teuthology.orchestra.run.smithi111:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-07T14:13:03.431 DEBUG:teuthology.misc:Transferring archived files from smithi145:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187/remote/smithi145 2024-04-07T14:13:03.431 DEBUG:teuthology.orchestra.run.smithi145:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-07T14:13:03.473 INFO:teuthology.task.internal:Removing archive directory... 2024-04-07T14:13:03.473 DEBUG:teuthology.orchestra.run.smithi111:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-07T14:13:03.475 DEBUG:teuthology.orchestra.run.smithi145:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-07T14:13:03.523 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-07T14:13:03.538 INFO:teuthology.task.internal:Not uploading archives. 2024-04-07T14:13:03.538 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-07T14:13:03.552 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-07T14:13:03.552 DEBUG:teuthology.orchestra.run.smithi111:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-07T14:13:03.553 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.01 MB/sec execute 145 sec latency 127976.047 ms 2024-04-07T14:13:03.554 DEBUG:teuthology.orchestra.run.smithi145:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-07T14:13:03.567 INFO:teuthology.orchestra.run.smithi111.stdout: 265306 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 7 14:13 /home/ubuntu/cephtest 2024-04-07T14:13:03.578 INFO:teuthology.orchestra.run.smithi145.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 7 14:13 /home/ubuntu/cephtest 2024-04-07T14:13:03.580 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-07T14:13:03.591 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-07T14:13:03.641 INFO:teuthology.nuke:Checking targets against current locks 2024-04-07T14:13:03.671 DEBUG:teuthology.nuke:shortname: smithi111 2024-04-07T14:13:03.671 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T14:13:03.695 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi111.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 13:46:50.584020', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBPXgwrwNSKGT7cIlN4zgFeJ2gcVfHjH+x7d72ytA5WhkIrhcWKLrSh033S9hw/RA6YjPdofi4DGFrdisK3rGXPw='} 2024-04-07T14:13:03.701 DEBUG:teuthology.nuke:shortname: smithi145 2024-04-07T14:13:03.702 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-07T14:13:03.726 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi145.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-04-01_20:08:13-orch-main-distro-default-smithi/7634187', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-07 13:46:50.582692', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOMaXteOlXjxMKd/pO4HJZvN7j53JHeyHukDvNFO3KA2Yi1tjH7Vf2qhr+59SY1gOG/+0oFi5qqciqug6vqo7dA='} 2024-04-07T14:13:03.761 INFO:teuthology.orchestra.console.smithi111:Power off 2024-04-07T14:13:03.762 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-07T14:13:03.800 INFO:teuthology.orchestra.console.smithi145:Power off 2024-04-07T14:13:03.801 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-07T14:13:03.809 DEBUG:teuthology.orchestra.console.smithi111:power off output: Chassis Power Control: Down/Off 2024-04-07T14:13:03.810 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:03.824 DEBUG:teuthology.orchestra.console.smithi145:power off output: Chassis Power Control: Down/Off 2024-04-07T14:13:03.825 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:03.844 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-07T14:13:03.851 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-07T14:13:04.546 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 2.00 MB/sec execute 146 sec latency 128976.147 ms 2024-04-07T14:13:05.546 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 1.98 MB/sec execute 147 sec latency 129976.245 ms 2024-04-07T14:13:06.546 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 1.97 MB/sec execute 148 sec latency 130976.319 ms 2024-04-07T14:13:07.546 INFO:teuthology.orchestra.run.smithi111.stdout: 5 89136 1.96 MB/sec execute 149 sec latency 131976.420 ms 2024-04-07T14:13:07.796 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-07T14:13:07.845 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:07.854 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:07.956 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-07T14:13:07.966 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-07T14:13:11.960 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:11.968 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:12.070 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is on 2024-04-07T14:13:12.077 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is on 2024-04-07T14:13:16.074 DEBUG:teuthology.orchestra.console.smithi111:pexpect command: ipmitool -H smithi111.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:16.083 DEBUG:teuthology.orchestra.console.smithi145:pexpect command: ipmitool -H smithi145.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-07T14:13:16.185 DEBUG:teuthology.orchestra.console.smithi111:check power output: Chassis Power is off 2024-04-07T14:13:16.195 DEBUG:teuthology.orchestra.console.smithi145:check power output: Chassis Power is off 2024-04-07T14:13:16.287 INFO:teuthology.orchestra.console.smithi111:Power off completed 2024-04-07T14:13:16.296 INFO:teuthology.orchestra.console.smithi145:Power off completed 2024-04-07T14:13:16.401 INFO:teuthology.run:Summary data: description: orch/cephadm/mgr-nfs-upgrade/{0-centos_9.stream 1-bootstrap/17.2.0 1-start 2-nfs 3-upgrade-with-workload 4-final} duration: 1015.0311422348022 failure_reason: 'Command failed on smithi111 with status 22: "sudo /home/ubuntu/cephtest/cephadm --image quay.io/ceph/ceph:v17.2.0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2bfa0756-f4e7-11ee-b649-cb9ed24678a4 -e sha1=e5c885fe95ee982ff7b45214daa61b1a6d09d57d -- bash -c ''ceph orch ps''"' owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=76b7dc0fd9b34ae88efcb444ac92ff2d status: fail success: false 2024-04-07T14:13:16.402 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-07T14:13:16.496 INFO:teuthology.run:FAIL