2023-10-03T06:56:31.287 INFO:root:teuthology version: 0.0.1.dev194+g54e62bc 2023-10-03T06:56:31.287 DEBUG:teuthology.run:Teuthology command: teuthology --description rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} --verbose --name bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi --archive /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514 --owner scheduled_bdavidov@teuthology -- /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/orig.config.yaml 2023-10-03T06:56:31.317 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T06:56:31.403 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514 branch: wip-barakda-cephadm-nvmeof description: rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} email: null first_in_suite: false job_id: '7409514' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - machine: cpus: 1 disk: 40 ram: 8000 volumes: count: 4 size: 30 os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-barakda-cephadm-nvmeof ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: bcea8c48517ff154f392584022efa0718f51c9f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: bcea8c48517ff154f392584022efa0718f51c9f2 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-nvmetest-teuthology sha1: b6b4a26de3cb4337d4479f11f356af936649f829 owner: scheduled_bdavidov@teuthology priority: 80 repo: https://git.ceph.com/ceph-ci.git roles: - - host.a - mon.a - mgr.x - osd.0 - osd.1 - client.0 - - host.b - mon.b - osd.2 - osd.3 - osd.4 - client.1 - - mon.c - osd.5 - osd.6 - osd.7 - client.2 seed: 4253 sha1: bcea8c48517ff154f392584022efa0718f51c9f2 sleep_before_teardown: 0 suite: rbd:nvmeof suite_branch: wip-nvmetest-teuthology suite_path: /home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa suite_relpath: qa suite_repo: https://github.com/barakda/ceph suite_sha1: b6b4a26de3cb4337d4479f11f356af936649f829 targets: smithi031.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJc0g7r8WVMYr3AaaywfLJYIjVH44RBAwx116cDxldOxZBLOdm1Ru8mwFiIjNFf2ZFRQUuRhpQm71jFXkvcdW8= smithi114.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCaaEs8/rx7U0jxdpTvEWhxgBfNhU/ko2AJyjCIlqYMQmHaSLe90uVGdhnF9cOuGFEPmn0xb0a+qqw9Xwya9rHw= smithi139.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKidOfiR9a47mR8/jcWpflnygMci/DvOOBqb3vVe0bHirAMu4JiWggY/PYr/YCkwyZcYSR8uLqzCstac9NbhIaY= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - sudo sed -i 's/runtime = "runc"/#runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/#runtime = "crun"/runtime = "crun"/g' /usr/share/containers/containers.conf - cephadm: null - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph osd lspools - ceph osd pool create mypool - ceph osd pool application enable mypool rbd - ceph osd lspools - ceph osd pool ls detail - ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/barakda1/nvmeof:0.0.3 - ceph config get mgr mgr/cephadm/container_image_nvmeof - ceph orch apply nvmeof mypool --placement="1 $(hostname)" - ceph orch ps --refresh - ceph versions - ceph -s - ceph orch ls - rbd create mypool/myimage --size 8Gi - rbd ls mypool - 'set -ex # cat /proc/cpuinfo ' - workunit: clients: client.0: - cephadm/test_nvme_etc_hosts.sh - cephadm.wait_for_service: service: nvmeof.mypool - exec: host.a: - 'set -ex # # print nvmeof start up logs s=$(systemctl list-units | grep nvmeof.mypool | awk ''{print $1}'') PAGER=cat journalctl -u $s ' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - ceph osd lspools teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 54e62bcbac4e53d9685e08328b790d3b20d71cae timestamp: 2023-10-03_06:50:26 tube: smithi use_shaman: true user: bdavidov verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2830 2023-10-03T06:56:31.403 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa; will attempt to use it 2023-10-03T06:56:31.404 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa/tasks 2023-10-03T06:56:31.404 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-10-03T06:56:31.406 INFO:teuthology.task.internal:Checking packages... 2023-10-03T06:56:31.426 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'bcea8c48517ff154f392584022efa0718f51c9f2' 2023-10-03T06:56:31.427 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-10-03T06:56:31.427 INFO:teuthology.packaging:ref: None 2023-10-03T06:56:31.427 INFO:teuthology.packaging:tag: None 2023-10-03T06:56:31.427 INFO:teuthology.packaging:branch: wip-barakda-cephadm-nvmeof 2023-10-03T06:56:31.427 INFO:teuthology.packaging:sha1: bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-03T06:56:31.427 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-barakda-cephadm-nvmeof 2023-10-03T06:56:31.676 INFO:teuthology.task.internal:Found packages for ceph version 18.0.0-5817.gbcea8c48 2023-10-03T06:56:31.676 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-10-03T06:56:31.683 INFO:teuthology.task.internal:no buildpackages task found 2023-10-03T06:56:31.683 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-10-03T06:56:31.725 INFO:teuthology.task.internal:Saving configuration 2023-10-03T06:56:31.735 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-10-03T06:56:31.758 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T06:56:31.782 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.388335', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJc0g7r8WVMYr3AaaywfLJYIjVH44RBAwx116cDxldOxZBLOdm1Ru8mwFiIjNFf2ZFRQUuRhpQm71jFXkvcdW8='} 2023-10-03T06:56:31.803 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.386631', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCaaEs8/rx7U0jxdpTvEWhxgBfNhU/ko2AJyjCIlqYMQmHaSLe90uVGdhnF9cOuGFEPmn0xb0a+qqw9Xwya9rHw='} 2023-10-03T06:56:31.827 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.389811', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKidOfiR9a47mR8/jcWpflnygMci/DvOOBqb3vVe0bHirAMu4JiWggY/PYr/YCkwyZcYSR8uLqzCstac9NbhIaY='} 2023-10-03T06:56:31.828 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-10-03T06:56:31.834 INFO:teuthology.task.internal:roles: ubuntu@smithi031.front.sepia.ceph.com - ['host.a', 'mon.a', 'mgr.x', 'osd.0', 'osd.1', 'client.0'] 2023-10-03T06:56:31.834 INFO:teuthology.task.internal:roles: ubuntu@smithi114.front.sepia.ceph.com - ['host.b', 'mon.b', 'osd.2', 'osd.3', 'osd.4', 'client.1'] 2023-10-03T06:56:31.834 INFO:teuthology.task.internal:roles: ubuntu@smithi139.front.sepia.ceph.com - ['mon.c', 'osd.5', 'osd.6', 'osd.7', 'client.2'] 2023-10-03T06:56:31.834 INFO:teuthology.run_tasks:Running task console_log... 2023-10-03T06:56:31.948 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fd7b953ab80>, signals=[15]) 2023-10-03T06:56:31.948 INFO:teuthology.run_tasks:Running task internal.connect... 2023-10-03T06:56:31.954 INFO:teuthology.task.internal:Opening connections... 2023-10-03T06:56:31.955 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T06:56:31.956 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:56:32.025 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi114.front.sepia.ceph.com 2023-10-03T06:56:32.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:56:32.091 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi139.front.sepia.ceph.com 2023-10-03T06:56:32.092 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:56:32.160 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-10-03T06:56:32.167 DEBUG:teuthology.orchestra.run.smithi031:> uname -m 2023-10-03T06:56:32.191 INFO:teuthology.orchestra.run.smithi031.stdout:x86_64 2023-10-03T06:56:32.191 DEBUG:teuthology.orchestra.run.smithi031:> cat /etc/os-release 2023-10-03T06:56:32.253 INFO:teuthology.orchestra.run.smithi031.stdout:NAME="CentOS Stream" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION="8" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:ID="centos" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:ID_LIKE="rhel fedora" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:VERSION_ID="8" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:PLATFORM_ID="platform:el8" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:ANSI_COLOR="0;31" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:HOME_URL="https://centos.org/" 2023-10-03T06:56:32.254 INFO:teuthology.orchestra.run.smithi031.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-03T06:56:32.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-03T06:56:32.255 INFO:teuthology.orchestra.run.smithi031.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-03T06:56:32.255 INFO:teuthology.lock.ops:Updating smithi031.front.sepia.ceph.com on lock server 2023-10-03T06:56:32.290 DEBUG:teuthology.orchestra.run.smithi114:> uname -m 2023-10-03T06:56:32.313 INFO:teuthology.orchestra.run.smithi114.stdout:x86_64 2023-10-03T06:56:32.313 DEBUG:teuthology.orchestra.run.smithi114:> cat /etc/os-release 2023-10-03T06:56:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:NAME="CentOS Stream" 2023-10-03T06:56:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION="8" 2023-10-03T06:56:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:ID="centos" 2023-10-03T06:56:32.374 INFO:teuthology.orchestra.run.smithi114.stdout:ID_LIKE="rhel fedora" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:VERSION_ID="8" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:PLATFORM_ID="platform:el8" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:ANSI_COLOR="0;31" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:HOME_URL="https://centos.org/" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-03T06:56:32.375 INFO:teuthology.orchestra.run.smithi114.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-03T06:56:32.376 INFO:teuthology.lock.ops:Updating smithi114.front.sepia.ceph.com on lock server 2023-10-03T06:56:32.398 DEBUG:teuthology.orchestra.run.smithi139:> uname -m 2023-10-03T06:56:32.421 INFO:teuthology.orchestra.run.smithi139.stdout:x86_64 2023-10-03T06:56:32.421 DEBUG:teuthology.orchestra.run.smithi139:> cat /etc/os-release 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:NAME="CentOS Stream" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION="8" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:ID="centos" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:ID_LIKE="rhel fedora" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:VERSION_ID="8" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:PLATFORM_ID="platform:el8" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:PRETTY_NAME="CentOS Stream 8" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:ANSI_COLOR="0;31" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-10-03T06:56:32.482 INFO:teuthology.orchestra.run.smithi139.stdout:HOME_URL="https://centos.org/" 2023-10-03T06:56:32.483 INFO:teuthology.orchestra.run.smithi139.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-10-03T06:56:32.483 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-10-03T06:56:32.483 INFO:teuthology.orchestra.run.smithi139.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-10-03T06:56:32.483 INFO:teuthology.lock.ops:Updating smithi139.front.sepia.ceph.com on lock server 2023-10-03T06:56:32.507 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-10-03T06:56:32.523 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-10-03T06:56:32.542 INFO:teuthology.task.internal:Checking for old test directory... 2023-10-03T06:56:32.543 DEBUG:teuthology.orchestra.run.smithi031:> test '!' -e /home/ubuntu/cephtest 2023-10-03T06:56:32.545 DEBUG:teuthology.orchestra.run.smithi114:> test '!' -e /home/ubuntu/cephtest 2023-10-03T06:56:32.547 DEBUG:teuthology.orchestra.run.smithi139:> test '!' -e /home/ubuntu/cephtest 2023-10-03T06:56:32.567 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-10-03T06:56:32.573 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-10-03T06:56:32.573 DEBUG:teuthology.orchestra.run.smithi031:> test -z $(ls -A /var/lib/ceph) 2023-10-03T06:56:32.607 DEBUG:teuthology.orchestra.run.smithi114:> test -z $(ls -A /var/lib/ceph) 2023-10-03T06:56:32.609 DEBUG:teuthology.orchestra.run.smithi139:> test -z $(ls -A /var/lib/ceph) 2023-10-03T06:56:32.648 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-10-03T06:56:32.722 INFO:teuthology.run_tasks:Running task kernel... 2023-10-03T06:56:32.736 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-10-03T06:56:32.737 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-10-03T06:56:32.737 DEBUG:teuthology.orchestra.run.smithi031:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T06:56:32.738 DEBUG:teuthology.orchestra.run.smithi114:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T06:56:32.738 DEBUG:teuthology.orchestra.run.smithi139:> test -f /run/.containerenv -o -f /.dockerenv 2023-10-03T06:56:32.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:56:32.758 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-10-03T06:56:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:56:32.759 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-03T06:56:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:56:32.761 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2023-10-03T06:56:32.819 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-489.el8.x86_64 2023-10-03T06:56:32.820 INFO:teuthology.task.kernel:Running kernel on smithi114: 4.18.0-489.el8.x86_64 2023-10-03T06:56:32.820 DEBUG:teuthology.orchestra.run.smithi114:> sudo yum install -y kernel 2023-10-03T06:56:32.822 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-489.el8.x86_64 2023-10-03T06:56:32.822 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-489.el8.x86_64 2023-10-03T06:56:32.822 INFO:teuthology.task.kernel:Running kernel on smithi139: 4.18.0-489.el8.x86_64 2023-10-03T06:56:32.822 DEBUG:teuthology.orchestra.run.smithi139:> sudo yum install -y kernel 2023-10-03T06:56:32.822 INFO:teuthology.task.kernel:Running kernel on smithi031: 4.18.0-489.el8.x86_64 2023-10-03T06:56:32.823 DEBUG:teuthology.orchestra.run.smithi031:> sudo yum install -y kernel 2023-10-03T06:56:37.192 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 8 - AppStream 17 MB/s | 33 MB 00:01 2023-10-03T06:56:37.301 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - AppStream 18 MB/s | 33 MB 00:01 2023-10-03T06:56:37.470 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 8 - AppStream 17 MB/s | 33 MB 00:01 2023-10-03T06:56:44.520 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - BaseOS 41 MB/s | 48 MB 00:01 2023-10-03T06:56:44.633 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 8 - BaseOS 49 MB/s | 48 MB 00:00 2023-10-03T06:56:44.836 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 8 - BaseOS 42 MB/s | 48 MB 00:01 2023-10-03T06:56:51.692 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras 52 kB/s | 18 kB 00:00 2023-10-03T06:56:52.019 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 8 - Extras 53 kB/s | 18 kB 00:00 2023-10-03T06:56:52.336 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 8 - Extras 49 kB/s | 18 kB 00:00 2023-10-03T06:56:52.376 INFO:teuthology.orchestra.run.smithi114.stdout:CentOS Stream 8 - Extras common packages 14 kB/s | 6.9 kB 00:00 2023-10-03T06:56:52.611 INFO:teuthology.orchestra.run.smithi139.stdout:CentOS Stream 8 - Extras common packages 17 kB/s | 6.9 kB 00:00 2023-10-03T06:56:52.761 INFO:teuthology.orchestra.run.smithi114.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-10-03T06:56:52.913 INFO:teuthology.orchestra.run.smithi139.stdout:Copr repo for python3-asyncssh owned by ceph 23 kB/s | 3.5 kB 00:00 2023-10-03T06:56:53.075 INFO:teuthology.orchestra.run.smithi031.stdout:CentOS Stream 8 - Extras common packages 13 kB/s | 6.9 kB 00:00 2023-10-03T06:56:53.401 INFO:teuthology.orchestra.run.smithi031.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-10-03T06:56:53.788 INFO:teuthology.orchestra.run.smithi139.stdout:Extra Packages for Enterprise Linux 22 MB/s | 16 MB 00:00 2023-10-03T06:56:54.694 INFO:teuthology.orchestra.run.smithi031.stdout:Extra Packages for Enterprise Linux 14 MB/s | 16 MB 00:01 2023-10-03T06:56:55.528 INFO:teuthology.orchestra.run.smithi114.stdout:Extra Packages for Enterprise Linux 6.1 MB/s | 16 MB 00:02 2023-10-03T06:56:57.697 INFO:teuthology.orchestra.run.smithi139.stdout:lab-extras 343 kB/s | 24 kB 00:00 2023-10-03T06:56:58.677 INFO:teuthology.orchestra.run.smithi031.stdout:lab-extras 576 kB/s | 24 kB 00:00 2023-10-03T06:56:59.320 INFO:teuthology.orchestra.run.smithi114.stdout:lab-extras 298 kB/s | 24 kB 00:00 2023-10-03T06:57:01.171 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-03T06:57:01.171 INFO:teuthology.orchestra.run.smithi139.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-03T06:57:01.417 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-10-03T06:57:01.418 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T06:57:01.418 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-10-03T06:57:01.418 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T06:57:01.419 INFO:teuthology.orchestra.run.smithi139.stdout:Installing: 2023-10-03T06:57:01.419 INFO:teuthology.orchestra.run.smithi139.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-03T06:57:01.419 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-10-03T06:57:01.419 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-03T06:57:01.420 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-03T06:57:01.420 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:01.420 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-10-03T06:57:01.420 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T06:57:01.420 INFO:teuthology.orchestra.run.smithi139.stdout:Install 3 Packages 2023-10-03T06:57:01.421 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:01.421 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 88 M 2023-10-03T06:57:01.422 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 96 M 2023-10-03T06:57:01.422 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-10-03T06:57:02.015 INFO:teuthology.orchestra.run.smithi139.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 24 MB/s | 10 MB 00:00 2023-10-03T06:57:02.363 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-03T06:57:02.364 INFO:teuthology.orchestra.run.smithi031.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-03T06:57:02.626 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T06:57:02.627 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T06:57:02.627 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-03T06:57:02.628 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T06:57:02.628 INFO:teuthology.orchestra.run.smithi031.stdout:Installing: 2023-10-03T06:57:02.628 INFO:teuthology.orchestra.run.smithi031.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-03T06:57:02.628 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T06:57:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-03T06:57:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-03T06:57:02.629 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:02.629 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T06:57:02.629 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T06:57:02.630 INFO:teuthology.orchestra.run.smithi031.stdout:Install 3 Packages 2023-10-03T06:57:02.630 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:02.631 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 88 M 2023-10-03T06:57:02.631 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 96 M 2023-10-03T06:57:02.631 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T06:57:02.640 INFO:teuthology.orchestra.run.smithi139.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 34 MB/s | 35 MB 00:01 2023-10-03T06:57:02.827 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-10-03T06:57:02.828 INFO:teuthology.orchestra.run.smithi114.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-10-03T06:57:03.074 INFO:teuthology.orchestra.run.smithi139.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 29 MB/s | 43 MB 00:01 2023-10-03T06:57:03.074 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-10-03T06:57:03.074 INFO:teuthology.orchestra.run.smithi139.stdout:Total 53 MB/s | 88 MB 00:01 2023-10-03T06:57:03.095 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-10-03T06:57:03.096 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T06:57:03.096 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2023-10-03T06:57:03.096 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T06:57:03.097 INFO:teuthology.orchestra.run.smithi114.stdout:Installing: 2023-10-03T06:57:03.097 INFO:teuthology.orchestra.run.smithi114.stdout: kernel x86_64 4.18.0-514.el8 baseos 10 M 2023-10-03T06:57:03.097 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2023-10-03T06:57:03.097 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-core x86_64 4.18.0-514.el8 baseos 43 M 2023-10-03T06:57:03.098 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules x86_64 4.18.0-514.el8 baseos 35 M 2023-10-03T06:57:03.098 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:03.098 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2023-10-03T06:57:03.098 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T06:57:03.099 INFO:teuthology.orchestra.run.smithi114.stdout:Install 3 Packages 2023-10-03T06:57:03.099 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:03.100 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 88 M 2023-10-03T06:57:03.100 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 96 M 2023-10-03T06:57:03.100 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2023-10-03T06:57:03.144 INFO:teuthology.orchestra.run.smithi031.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 21 MB/s | 10 MB 00:00 2023-10-03T06:57:03.743 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-10-03T06:57:03.745 INFO:teuthology.orchestra.run.smithi031.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 33 MB/s | 35 MB 00:01 2023-10-03T06:57:03.880 INFO:teuthology.orchestra.run.smithi114.stdout:(1/3): kernel-4.18.0-514.el8.x86_64.rpm 27 MB/s | 10 MB 00:00 2023-10-03T06:57:04.182 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-10-03T06:57:04.182 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-10-03T06:57:04.188 INFO:teuthology.orchestra.run.smithi031.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 28 MB/s | 43 MB 00:01 2023-10-03T06:57:04.188 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T06:57:04.188 INFO:teuthology.orchestra.run.smithi031.stdout:Total 56 MB/s | 88 MB 00:01 2023-10-03T06:57:04.521 INFO:teuthology.orchestra.run.smithi114.stdout:(2/3): kernel-modules-4.18.0-514.el8.x86_64.rpm 35 MB/s | 35 MB 00:01 2023-10-03T06:57:04.894 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T06:57:04.955 INFO:teuthology.orchestra.run.smithi114.stdout:(3/3): kernel-core-4.18.0-514.el8.x86_64.rpm 29 MB/s | 43 MB 00:01 2023-10-03T06:57:04.955 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2023-10-03T06:57:04.956 INFO:teuthology.orchestra.run.smithi114.stdout:Total 47 MB/s | 88 MB 00:01 2023-10-03T06:57:05.323 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T06:57:05.323 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T06:57:05.398 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-10-03T06:57:05.425 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-10-03T06:57:05.626 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2023-10-03T06:57:06.057 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2023-10-03T06:57:06.057 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2023-10-03T06:57:06.609 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T06:57:06.629 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T06:57:07.289 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2023-10-03T06:57:07.316 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2023-10-03T06:57:08.215 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-10-03T06:57:08.859 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:09.545 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T06:57:10.049 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2023-10-03T06:57:10.079 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:10.639 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:12.441 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:12.859 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:13.792 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:14.183 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:14.252 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:14.638 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:16.235 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:16.548 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:17.801 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:18.015 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:18.117 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:18.335 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:38.185 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:38.208 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:38.562 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:38.562 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:38.563 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:39.355 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:39.356 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-10-03T06:57:39.487 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:39.741 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:39.742 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:39.742 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-03T06:57:39.742 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-03T06:57:39.743 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-03T06:57:39.743 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2023-10-03T06:57:39.743 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi139.front.sepia.ceph.com, path=None, version=distro) 2023-10-03T06:57:39.743 DEBUG:teuthology.orchestra.run.smithi139:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:39.951 INFO:teuthology.orchestra.run.smithi139.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:39.951 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:39.952 DEBUG:teuthology.orchestra.run.smithi139:> sudo rpm -qi grub2-tools 2023-10-03T06:57:39.985 INFO:teuthology.orchestra.run.smithi139.stdout:Name : grub2-tools 2023-10-03T06:57:39.985 INFO:teuthology.orchestra.run.smithi139.stdout:Epoch : 1 2023-10-03T06:57:39.986 INFO:teuthology.orchestra.run.smithi139.stdout:Version : 2.02 2023-10-03T06:57:39.986 INFO:teuthology.orchestra.run.smithi139.stdout:Release : 129.el8 2023-10-03T06:57:39.986 INFO:teuthology.orchestra.run.smithi139.stdout:Architecture: x86_64 2023-10-03T06:57:39.986 INFO:teuthology.orchestra.run.smithi139.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-03T06:57:39.986 INFO:teuthology.orchestra.run.smithi139.stdout:Group : System Environment/Base 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Size : 9265586 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:License : GPLv3+ 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-03T06:57:39.987 INFO:teuthology.orchestra.run.smithi139.stdout:Relocations : (not relocatable) 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:Packager : CentOS BuildSystem 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:Vendor : CentOS 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:Summary : Support tools for GRUB. 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:Description : 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-03T06:57:39.988 INFO:teuthology.orchestra.run.smithi139.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-03T06:57:39.989 INFO:teuthology.orchestra.run.smithi139.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-03T06:57:39.989 INFO:teuthology.orchestra.run.smithi139.stdout:hardware devices. 2023-10-03T06:57:39.989 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T06:57:39.989 INFO:teuthology.orchestra.run.smithi139.stdout:This subpackage provides tools for support of all platforms. 2023-10-03T06:57:39.990 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-03T06:57:39.990 INFO:teuthology.task.kernel:Updating grub on smithi139 to boot 4.18.0-514.el8.x86_64 2023-10-03T06:57:39.990 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-03T06:57:40.037 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:40.061 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:40.405 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:41.193 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:41.193 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:41.193 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2023-10-03T06:57:41.193 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-03T06:57:41.193 INFO:teuthology.orchestra.run.smithi114.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-03T06:57:41.194 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:41.194 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-10-03T06:57:41.303 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-core-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:41.326 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:41.344 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:41.559 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:41.559 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:41.560 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-03T06:57:41.560 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-03T06:57:41.560 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-03T06:57:41.560 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-10-03T06:57:41.560 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi114.front.sepia.ceph.com, path=None, version=distro) 2023-10-03T06:57:41.560 DEBUG:teuthology.orchestra.run.smithi114:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:41.687 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: kernel-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:41.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-4.18.0-514.el8.x86_64 1/3 2023-10-03T06:57:41.688 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-core-4.18.0-514.el8.x86_64 2/3 2023-10-03T06:57:41.766 INFO:teuthology.orchestra.run.smithi114.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:41.767 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:41.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo rpm -qi grub2-tools 2023-10-03T06:57:41.791 INFO:teuthology.orchestra.run.smithi139.stderr:Generating grub configuration file ... 2023-10-03T06:57:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:Name : grub2-tools 2023-10-03T06:57:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:Epoch : 1 2023-10-03T06:57:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:Version : 2.02 2023-10-03T06:57:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:Release : 129.el8 2023-10-03T06:57:41.801 INFO:teuthology.orchestra.run.smithi114.stdout:Architecture: x86_64 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Group : System Environment/Base 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Size : 9265586 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:License : GPLv3+ 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-03T06:57:41.802 INFO:teuthology.orchestra.run.smithi114.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Relocations : (not relocatable) 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Packager : CentOS BuildSystem 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Vendor : CentOS 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Summary : Support tools for GRUB. 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:Description : 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:41.803 INFO:teuthology.orchestra.run.smithi114.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-03T06:57:41.804 INFO:teuthology.orchestra.run.smithi114.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-03T06:57:41.804 INFO:teuthology.orchestra.run.smithi114.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-03T06:57:41.804 INFO:teuthology.orchestra.run.smithi114.stdout:hardware devices. 2023-10-03T06:57:41.804 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T06:57:41.804 INFO:teuthology.orchestra.run.smithi114.stdout:This subpackage provides tools for support of all platforms. 2023-10-03T06:57:41.805 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-03T06:57:41.805 INFO:teuthology.task.kernel:Updating grub on smithi114 to boot 4.18.0-514.el8.x86_64 2023-10-03T06:57:41.806 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-03T06:57:42.411 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : kernel-modules-4.18.0-514.el8.x86_64 3/3 2023-10-03T06:57:42.411 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:42.411 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T06:57:42.411 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-4.18.0-514.el8.x86_64 kernel-core-4.18.0-514.el8.x86_64 2023-10-03T06:57:42.411 INFO:teuthology.orchestra.run.smithi031.stdout: kernel-modules-4.18.0-514.el8.x86_64 2023-10-03T06:57:42.412 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:42.412 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T06:57:42.548 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:42.820 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:42.820 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:42.821 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-514.el8.x86_64 2023-10-03T06:57:42.821 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-10-03T06:57:42.821 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-10-03T06:57:42.821 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-10-03T06:57:42.821 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi031.front.sepia.ceph.com, path=None, version=distro) 2023-10-03T06:57:42.821 DEBUG:teuthology.orchestra.run.smithi031:> rpm -q kernel | sort -rV | head -n 1 2023-10-03T06:57:43.039 INFO:teuthology.orchestra.run.smithi031.stdout:kernel-4.18.0-514.el8.x86_64 2023-10-03T06:57:43.040 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-514.el8.x86_64 2023-10-03T06:57:43.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo rpm -qi grub2-tools 2023-10-03T06:57:43.075 INFO:teuthology.orchestra.run.smithi031.stdout:Name : grub2-tools 2023-10-03T06:57:43.075 INFO:teuthology.orchestra.run.smithi031.stdout:Epoch : 1 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Version : 2.02 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Release : 129.el8 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Architecture: x86_64 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Group : System Environment/Base 2023-10-03T06:57:43.076 INFO:teuthology.orchestra.run.smithi031.stdout:Size : 9265586 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:License : GPLv3+ 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:Build Host : kbuilder.bsys.centos.org 2023-10-03T06:57:43.077 INFO:teuthology.orchestra.run.smithi031.stdout:Relocations : (not relocatable) 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:Packager : CentOS BuildSystem 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:Vendor : CentOS 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:URL : http://www.gnu.org/software/grub/ 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:Summary : Support tools for GRUB. 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:Description : 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:43.078 INFO:teuthology.orchestra.run.smithi031.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-10-03T06:57:43.079 INFO:teuthology.orchestra.run.smithi031.stdout:customizable bootloader with modular architecture. It supports a rich 2023-10-03T06:57:43.079 INFO:teuthology.orchestra.run.smithi031.stdout:variety of kernel formats, file systems, computer architectures and 2023-10-03T06:57:43.079 INFO:teuthology.orchestra.run.smithi031.stdout:hardware devices. 2023-10-03T06:57:43.079 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T06:57:43.079 INFO:teuthology.orchestra.run.smithi031.stdout:This subpackage provides tools for support of all platforms. 2023-10-03T06:57:43.080 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-10-03T06:57:43.080 INFO:teuthology.task.kernel:Updating grub on smithi031 to boot 4.18.0-514.el8.x86_64 2023-10-03T06:57:43.081 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-10-03T06:57:43.593 INFO:teuthology.orchestra.run.smithi114.stderr:Generating grub configuration file ... 2023-10-03T06:57:43.889 INFO:teuthology.orchestra.run.smithi139.stderr:done 2023-10-03T06:57:43.891 DEBUG:teuthology.orchestra.run.smithi139:> mktemp 2023-10-03T06:57:43.925 INFO:teuthology.orchestra.run.smithi139.stdout:/tmp/tmp.yRXDdylyI4 2023-10-03T06:57:43.925 DEBUG:teuthology.orchestra.run.smithi139:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yRXDdylyI4 2023-10-03T06:57:43.998 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0666 /tmp/tmp.yRXDdylyI4 2023-10-03T06:57:44.142 DEBUG:teuthology.orchestra.remote:smithi139:/tmp/tmp.yRXDdylyI4 is 6KB 2023-10-03T06:57:44.192 DEBUG:teuthology.orchestra.run.smithi139:> rm -fr /tmp/tmp.yRXDdylyI4 2023-10-03T06:57:44.209 DEBUG:teuthology.orchestra.run.smithi139:> sudo /bin/ls /boot/loader/entries || true 2023-10-03T06:57:44.274 INFO:teuthology.orchestra.run.smithi139.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-03T06:57:44.274 INFO:teuthology.orchestra.run.smithi139.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-03T06:57:44.275 INFO:teuthology.orchestra.run.smithi139.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-03T06:57:44.275 INFO:teuthology.orchestra.run.smithi139.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-03T06:57:44.276 DEBUG:teuthology.orchestra.run.smithi139:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-03T06:57:44.415 INFO:teuthology.orchestra.run.smithi031.stderr:Generating grub configuration file ... 2023-10-03T06:57:44.463 DEBUG:teuthology.orchestra.run.smithi139:> sudo shutdown -r now 2023-10-03T06:57:45.479 INFO:teuthology.orchestra.run.smithi114.stderr:done 2023-10-03T06:57:45.480 DEBUG:teuthology.orchestra.run.smithi114:> mktemp 2023-10-03T06:57:45.521 INFO:teuthology.orchestra.run.smithi114.stdout:/tmp/tmp.WCc9UQWJos 2023-10-03T06:57:45.521 DEBUG:teuthology.orchestra.run.smithi114:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.WCc9UQWJos 2023-10-03T06:57:45.603 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0666 /tmp/tmp.WCc9UQWJos 2023-10-03T06:57:45.748 DEBUG:teuthology.orchestra.remote:smithi114:/tmp/tmp.WCc9UQWJos is 6KB 2023-10-03T06:57:45.798 DEBUG:teuthology.orchestra.run.smithi114:> rm -fr /tmp/tmp.WCc9UQWJos 2023-10-03T06:57:45.816 DEBUG:teuthology.orchestra.run.smithi114:> sudo /bin/ls /boot/loader/entries || true 2023-10-03T06:57:45.913 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-03T06:57:45.913 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-03T06:57:45.913 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-03T06:57:45.913 INFO:teuthology.orchestra.run.smithi114.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-03T06:57:45.915 DEBUG:teuthology.orchestra.run.smithi114:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-03T06:57:46.034 DEBUG:teuthology.orchestra.run.smithi114:> sudo shutdown -r now 2023-10-03T06:57:46.902 INFO:teuthology.orchestra.run.smithi031.stderr:done 2023-10-03T06:57:46.903 DEBUG:teuthology.orchestra.run.smithi031:> mktemp 2023-10-03T06:57:46.931 INFO:teuthology.orchestra.run.smithi031.stdout:/tmp/tmp.tVSCx4R3Vo 2023-10-03T06:57:46.931 DEBUG:teuthology.orchestra.run.smithi031:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.tVSCx4R3Vo 2023-10-03T06:57:47.003 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0666 /tmp/tmp.tVSCx4R3Vo 2023-10-03T06:57:47.171 DEBUG:teuthology.orchestra.remote:smithi031:/tmp/tmp.tVSCx4R3Vo is 6KB 2023-10-03T06:57:47.220 DEBUG:teuthology.orchestra.run.smithi031:> rm -fr /tmp/tmp.tVSCx4R3Vo 2023-10-03T06:57:47.238 DEBUG:teuthology.orchestra.run.smithi031:> sudo /bin/ls /boot/loader/entries || true 2023-10-03T06:57:47.316 INFO:teuthology.orchestra.run.smithi031.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-10-03T06:57:47.317 INFO:teuthology.orchestra.run.smithi031.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-10-03T06:57:47.317 INFO:teuthology.orchestra.run.smithi031.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-10-03T06:57:47.317 INFO:teuthology.orchestra.run.smithi031.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64.conf 2023-10-03T06:57:47.318 DEBUG:teuthology.orchestra.run.smithi031:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-514.el8.x86_64 2023-10-03T06:57:47.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo shutdown -r now 2023-10-03T06:58:14.471 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2023-10-03T06:58:14.473 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-10-03T06:58:14.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:58:16.039 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-10-03T06:58:16.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T06:58:16.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:58:17.475 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-10-03T06:58:17.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T06:58:17.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:58:50.619 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-10-03T06:58:59.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T06:58:59.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:59:14.491 DEBUG:teuthology.orchestra.remote:timed out 2023-10-03T06:59:17.481 DEBUG:teuthology.orchestra.remote:timed out 2023-10-03T06:59:18.010 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.114 2023-10-03T06:59:23.498 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-10-03T06:59:23.499 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:59:23.799 DEBUG:teuthology.orchestra.run.smithi139:> true 2023-10-03T06:59:24.220 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-10-03T06:59:24.220 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "4.18.0-514.el8.x86_64"... 2023-10-03T06:59:24.220 DEBUG:teuthology.orchestra.run.smithi139:> uname -r 2023-10-03T06:59:24.239 INFO:teuthology.orchestra.run.smithi139.stdout:4.18.0-514.el8.x86_64 2023-10-03T06:59:24.239 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-03T06:59:24.239 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-03T06:59:24.239 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-03T06:59:25.240 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2023-10-03T06:59:25.241 DEBUG:teuthology.orchestra.run.smithi139:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T06:59:25.312 INFO:teuthology.orchestra.run.smithi139.stdout:ttyS1 2023-10-03T06:59:25.325 DEBUG:teuthology.parallel:result is None 2023-10-03T06:59:26.483 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T06:59:26.484 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:59:26.858 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-03T06:59:27.162 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T06:59:27.162 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-514.el8.x86_64"... 2023-10-03T06:59:27.163 DEBUG:teuthology.orchestra.run.smithi031:> uname -r 2023-10-03T06:59:27.185 INFO:teuthology.orchestra.run.smithi031.stdout:4.18.0-514.el8.x86_64 2023-10-03T06:59:27.186 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-03T06:59:27.186 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-03T06:59:27.186 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-03T06:59:28.187 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-10-03T06:59:28.187 DEBUG:teuthology.orchestra.run.smithi031:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T06:59:28.263 INFO:teuthology.orchestra.run.smithi031.stdout:ttyS1 2023-10-03T06:59:28.277 DEBUG:teuthology.parallel:result is None 2023-10-03T06:59:30.011 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T06:59:30.011 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T06:59:30.359 DEBUG:teuthology.orchestra.run.smithi114:> true 2023-10-03T06:59:30.654 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T06:59:30.655 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-514.el8.x86_64"... 2023-10-03T06:59:30.655 DEBUG:teuthology.orchestra.run.smithi114:> uname -r 2023-10-03T06:59:30.672 INFO:teuthology.orchestra.run.smithi114.stdout:4.18.0-514.el8.x86_64 2023-10-03T06:59:30.672 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-514.el8.x86_64 vs 4.18.0-514.el8.x86_64 2023-10-03T06:59:30.673 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-10-03T06:59:30.673 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-10-03T06:59:31.674 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-10-03T06:59:31.675 DEBUG:teuthology.orchestra.run.smithi114:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-10-03T06:59:31.784 INFO:teuthology.orchestra.run.smithi114.stdout:ttyS1 2023-10-03T06:59:31.804 DEBUG:teuthology.parallel:result is None 2023-10-03T06:59:31.804 INFO:teuthology.run_tasks:Running task internal.base... 2023-10-03T06:59:31.814 INFO:teuthology.task.internal:Creating test directory... 2023-10-03T06:59:31.814 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T06:59:31.817 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T06:59:31.820 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-10-03T06:59:31.915 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-10-03T06:59:31.922 INFO:teuthology.run_tasks:Running task internal.archive... 2023-10-03T06:59:31.943 INFO:teuthology.task.internal:Creating archive directory... 2023-10-03T06:59:31.944 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T06:59:31.947 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T06:59:31.949 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-10-03T06:59:31.983 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-10-03T06:59:31.990 INFO:teuthology.task.internal:Enabling coredump saving... 2023-10-03T06:59:31.991 DEBUG:teuthology.orchestra.run.smithi031:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T06:59:32.017 DEBUG:teuthology.orchestra.run.smithi114:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T06:59:32.020 DEBUG:teuthology.orchestra.run.smithi139:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-10-03T06:59:32.048 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.053 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.055 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.058 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.064 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.068 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-10-03T06:59:32.069 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-10-03T06:59:32.077 INFO:teuthology.task.internal:Configuring sudo... 2023-10-03T06:59:32.078 DEBUG:teuthology.orchestra.run.smithi031:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T06:59:32.113 DEBUG:teuthology.orchestra.run.smithi114:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T06:59:32.116 DEBUG:teuthology.orchestra.run.smithi139:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-10-03T06:59:32.146 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-10-03T06:59:32.156 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-10-03T06:59:32.157 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T06:59:32.189 DEBUG:teuthology.orchestra.run.smithi114:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T06:59:32.192 DEBUG:teuthology.orchestra.run.smithi139:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-10-03T06:59:32.211 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:32.305 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:32.365 DEBUG:teuthology.orchestra.run.smithi031:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:32.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:32.467 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T06:59:32.467 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T06:59:32.542 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:32.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:32.692 DEBUG:teuthology.orchestra.run.smithi114:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:32.805 DEBUG:teuthology.orchestra.run.smithi114:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:33.096 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T06:59:33.096 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T06:59:33.642 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:33.694 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-10-03T06:59:33.744 DEBUG:teuthology.orchestra.run.smithi139:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:33.830 DEBUG:teuthology.orchestra.run.smithi139:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-10-03T06:59:33.873 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T06:59:33.873 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-10-03T06:59:33.940 DEBUG:teuthology.orchestra.run.smithi031:> sudo service rsyslog restart 2023-10-03T06:59:33.943 DEBUG:teuthology.orchestra.run.smithi114:> sudo service rsyslog restart 2023-10-03T06:59:33.982 DEBUG:teuthology.orchestra.run.smithi139:> sudo service rsyslog restart 2023-10-03T06:59:34.002 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T06:59:34.056 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T06:59:34.319 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T06:59:34.648 INFO:teuthology.run_tasks:Running task internal.timer... 2023-10-03T06:59:34.657 INFO:teuthology.task.internal:Starting timer... 2023-10-03T06:59:34.657 INFO:teuthology.run_tasks:Running task pcp... 2023-10-03T06:59:34.697 INFO:teuthology.run_tasks:Running task selinux... 2023-10-03T06:59:34.728 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-10-03T06:59:34.729 DEBUG:teuthology.orchestra.run.smithi031:> sudo service auditd rotate 2023-10-03T06:59:34.813 INFO:teuthology.orchestra.run.smithi031.stdout:Rotating logs: 2023-10-03T06:59:34.814 DEBUG:teuthology.orchestra.run.smithi114:> sudo service auditd rotate 2023-10-03T06:59:35.178 INFO:teuthology.orchestra.run.smithi114.stdout:Rotating logs: 2023-10-03T06:59:35.180 DEBUG:teuthology.orchestra.run.smithi139:> sudo service auditd rotate 2023-10-03T06:59:35.244 INFO:teuthology.orchestra.run.smithi139.stdout:Rotating logs: 2023-10-03T06:59:35.246 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-10-03T06:59:35.247 DEBUG:teuthology.orchestra.run.smithi031:> /usr/sbin/getenforce 2023-10-03T06:59:35.429 INFO:teuthology.orchestra.run.smithi031.stdout:Permissive 2023-10-03T06:59:35.429 DEBUG:teuthology.orchestra.run.smithi114:> /usr/sbin/getenforce 2023-10-03T06:59:35.454 INFO:teuthology.orchestra.run.smithi114.stdout:Permissive 2023-10-03T06:59:35.455 DEBUG:teuthology.orchestra.run.smithi139:> /usr/sbin/getenforce 2023-10-03T06:59:35.478 INFO:teuthology.orchestra.run.smithi139.stdout:Permissive 2023-10-03T06:59:35.478 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi031.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi114.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi139.front.sepia.ceph.com': 'permissive'} 2023-10-03T06:59:35.479 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T06:59:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:59:35.516 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T06:59:35.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:59:35.544 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T06:59:35.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T06:59:35.573 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-10-03T06:59:35.573 DEBUG:teuthology.orchestra.run.smithi031:> sudo /usr/sbin/setenforce permissive 2023-10-03T06:59:35.605 DEBUG:teuthology.orchestra.run.smithi114:> sudo /usr/sbin/setenforce permissive 2023-10-03T06:59:35.637 DEBUG:teuthology.orchestra.run.smithi139:> sudo /usr/sbin/setenforce permissive 2023-10-03T06:59:35.665 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-10-03T06:59:35.677 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-10-03T06:59:35.685 INFO:teuthology.repo_utils:Fetching main from origin 2023-10-03T06:59:35.862 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-10-03T06:59:35.904 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-10-03T06:59:35.906 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi031.front.sepia.ceph.com,smithi114.front.sepia.ceph.com,smithi139.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-10-03T07:07:23.537 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi031.front.sepia.ceph.com'), Remote(name='ubuntu@smithi114.front.sepia.ceph.com'), Remote(name='ubuntu@smithi139.front.sepia.ceph.com')] 2023-10-03T07:07:23.538 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T07:07:23.539 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi031.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T07:07:23.614 DEBUG:teuthology.orchestra.run.smithi031:> true 2023-10-03T07:07:23.690 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi031.front.sepia.ceph.com' 2023-10-03T07:07:23.691 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T07:07:23.691 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi114.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T07:07:23.758 DEBUG:teuthology.orchestra.run.smithi114:> true 2023-10-03T07:07:23.829 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi114.front.sepia.ceph.com' 2023-10-03T07:07:23.829 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-10-03T07:07:23.829 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi139.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-10-03T07:07:23.899 DEBUG:teuthology.orchestra.run.smithi139:> true 2023-10-03T07:07:23.973 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi139.front.sepia.ceph.com' 2023-10-03T07:07:23.973 INFO:teuthology.run_tasks:Running task clock... 2023-10-03T07:07:23.984 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-10-03T07:07:23.984 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T07:07:23.985 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:07:23.988 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T07:07:23.988 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:07:23.990 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-10-03T07:07:23.991 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:07:24.021 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-03T07:07:24.026 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-03T07:07:24.036 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-03T07:07:24.043 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-03T07:07:24.046 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-10-03T07:07:24.060 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-10-03T07:07:24.094 INFO:teuthology.orchestra.run.smithi114.stderr:sudo: ntpd: command not found 2023-10-03T07:07:24.106 INFO:teuthology.orchestra.run.smithi114.stdout:506 Cannot talk to daemon 2023-10-03T07:07:24.117 INFO:teuthology.orchestra.run.smithi031.stderr:sudo: ntpd: command not found 2023-10-03T07:07:24.120 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-03T07:07:24.129 INFO:teuthology.orchestra.run.smithi139.stderr:sudo: ntpd: command not found 2023-10-03T07:07:24.130 INFO:teuthology.orchestra.run.smithi031.stdout:506 Cannot talk to daemon 2023-10-03T07:07:24.133 INFO:teuthology.orchestra.run.smithi114.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-03T07:07:24.141 INFO:teuthology.orchestra.run.smithi139.stdout:506 Cannot talk to daemon 2023-10-03T07:07:24.146 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-03T07:07:24.155 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-10-03T07:07:24.162 INFO:teuthology.orchestra.run.smithi031.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-03T07:07:24.168 INFO:teuthology.orchestra.run.smithi139.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-10-03T07:07:24.199 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.203 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.231 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-10-03T07:07:24.232 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.235 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.237 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:07:24.238 INFO:teuthology.run_tasks:Running task pexec... 2023-10-03T07:07:24.247 INFO:teuthology.task.pexec:Executing custom commands... 2023-10-03T07:07:24.247 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T07:07:24.247 DEBUG:teuthology.orchestra.run.smithi031:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-03T07:07:24.248 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi114.front.sepia.ceph.com 2023-10-03T07:07:24.248 DEBUG:teuthology.orchestra.run.smithi114:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-03T07:07:24.248 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi139.front.sepia.ceph.com 2023-10-03T07:07:24.248 DEBUG:teuthology.orchestra.run.smithi139:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-10-03T07:07:25.124 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:19 ago on Tue 03 Oct 2023 07:05:05 AM UTC. 2023-10-03T07:07:25.161 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:19 ago on Tue 03 Oct 2023 07:05:05 AM UTC. 2023-10-03T07:07:25.292 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:38 ago on Tue 03 Oct 2023 07:05:47 AM UTC. 2023-10-03T07:07:27.116 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-10-03T07:07:27.116 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout: Package Architecture Version Repository Size 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout:Resetting modules: 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:27.117 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:27.187 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout: Package Architecture Version Repository Size 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout:Resetting modules: 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools 2023-10-03T07:07:27.188 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:27.189 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-10-03T07:07:27.189 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:27.189 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:27.365 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-10-03T07:07:27.443 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-10-03T07:07:27.534 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T07:07:27.535 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:27.535 INFO:teuthology.orchestra.run.smithi031.stdout: Package Architecture Version Repository Size 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout:Resetting modules: 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:27.536 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:27.826 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T07:07:28.306 INFO:teuthology.orchestra.run.smithi114.stdout:Last metadata expiration check: 0:02:23 ago on Tue 03 Oct 2023 07:05:05 AM UTC. 2023-10-03T07:07:28.377 INFO:teuthology.orchestra.run.smithi139.stdout:Last metadata expiration check: 0:02:23 ago on Tue 03 Oct 2023 07:05:05 AM UTC. 2023-10-03T07:07:28.919 INFO:teuthology.orchestra.run.smithi031.stdout:Last metadata expiration check: 0:01:41 ago on Tue 03 Oct 2023 07:05:47 AM UTC. 2023-10-03T07:07:30.326 INFO:teuthology.orchestra.run.smithi114.stdout:Dependencies resolved. 2023-10-03T07:07:30.378 INFO:teuthology.orchestra.run.smithi139.stdout:Dependencies resolved. 2023-10-03T07:07:30.481 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:30.481 INFO:teuthology.orchestra.run.smithi114.stdout: Package Arch Version Repository Size 2023-10-03T07:07:30.481 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:30.481 INFO:teuthology.orchestra.run.smithi114.stdout:Installing group/module packages: 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-03T07:07:30.482 INFO:teuthology.orchestra.run.smithi114.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout:Installing dependencies: 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout:Installing weak dependencies: 2023-10-03T07:07:30.483 INFO:teuthology.orchestra.run.smithi114.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout:Installing module profiles: 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools/common 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout:Enabling module streams: 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout: container-tools rhel8 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction Summary 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout:================================================================================ 2023-10-03T07:07:30.484 INFO:teuthology.orchestra.run.smithi114.stdout:Install 13 Packages 2023-10-03T07:07:30.485 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:30.485 INFO:teuthology.orchestra.run.smithi114.stdout:Total download size: 26 M 2023-10-03T07:07:30.486 INFO:teuthology.orchestra.run.smithi114.stdout:Installed size: 87 M 2023-10-03T07:07:30.486 INFO:teuthology.orchestra.run.smithi114.stdout:Downloading Packages: 2023-10-03T07:07:30.536 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:30.536 INFO:teuthology.orchestra.run.smithi139.stdout: Package Arch Version Repository Size 2023-10-03T07:07:30.536 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout:Installing group/module packages: 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-03T07:07:30.537 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout:Installing dependencies: 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout:Installing weak dependencies: 2023-10-03T07:07:30.538 INFO:teuthology.orchestra.run.smithi139.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout:Installing module profiles: 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools/common 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout:Enabling module streams: 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout: container-tools rhel8 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction Summary 2023-10-03T07:07:30.539 INFO:teuthology.orchestra.run.smithi139.stdout:================================================================================ 2023-10-03T07:07:30.540 INFO:teuthology.orchestra.run.smithi139.stdout:Install 13 Packages 2023-10-03T07:07:30.540 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:30.540 INFO:teuthology.orchestra.run.smithi139.stdout:Total download size: 26 M 2023-10-03T07:07:30.541 INFO:teuthology.orchestra.run.smithi139.stdout:Installed size: 87 M 2023-10-03T07:07:30.541 INFO:teuthology.orchestra.run.smithi139.stdout:Downloading Packages: 2023-10-03T07:07:31.117 INFO:teuthology.orchestra.run.smithi114.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.6 MB/s | 739 kB 00:00 2023-10-03T07:07:31.145 INFO:teuthology.orchestra.run.smithi031.stdout:Dependencies resolved. 2023-10-03T07:07:31.150 INFO:teuthology.orchestra.run.smithi114.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 2.1 MB/s | 1.0 MB 00:00 2023-10-03T07:07:31.160 INFO:teuthology.orchestra.run.smithi139.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.7 MB/s | 739 kB 00:00 2023-10-03T07:07:31.194 INFO:teuthology.orchestra.run.smithi139.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 2.1 MB/s | 1.0 MB 00:00 2023-10-03T07:07:31.201 INFO:teuthology.orchestra.run.smithi114.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 452 kB/s | 38 kB 00:00 2023-10-03T07:07:31.236 INFO:teuthology.orchestra.run.smithi139.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 502 kB/s | 38 kB 00:00 2023-10-03T07:07:31.243 INFO:teuthology.orchestra.run.smithi114.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.5 MB/s | 239 kB 00:00 2023-10-03T07:07:31.278 INFO:teuthology.orchestra.run.smithi139.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.8 MB/s | 239 kB 00:00 2023-10-03T07:07:31.320 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:31.320 INFO:teuthology.orchestra.run.smithi031.stdout: Package Arch Version Repository Size 2023-10-03T07:07:31.320 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:31.320 INFO:teuthology.orchestra.run.smithi031.stdout:Installing group/module packages: 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: aardvark-dns x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 1.0 M 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: buildah x86_64 1:1.31.3-1.module_el8+664+4072b3ae appstream 8.8 M 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: cockpit-podman noarch 75-1.module_el8+664+4072b3ae appstream 739 k 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: crun x86_64 1.8.7-1.module_el8+661+d1afb926 appstream 239 k 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: netavark x86_64 2:1.7.0-1.module_el8+487+8e42a277 appstream 3.7 M 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: python3-podman noarch 4.6.0-1.module_el8+582+a2558ae2 appstream 177 k 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: skopeo x86_64 2:1.13.3-1.module_el8+664+4072b3ae appstream 8.1 M 2023-10-03T07:07:31.321 INFO:teuthology.orchestra.run.smithi031.stdout: toolbox x86_64 0.0.99.4-5.module_el8+649+e2deac1d appstream 2.5 M 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout: udica noarch 0.2.6-20.module_el8+487+8e42a277 appstream 53 k 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout:Installing dependencies: 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout:Installing weak dependencies: 2023-10-03T07:07:31.322 INFO:teuthology.orchestra.run.smithi031.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout:Installing module profiles: 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools/common 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling module streams: 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout: container-tools rhel8 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction Summary 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout:================================================================================ 2023-10-03T07:07:31.323 INFO:teuthology.orchestra.run.smithi031.stdout:Install 13 Packages 2023-10-03T07:07:31.324 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:31.324 INFO:teuthology.orchestra.run.smithi031.stdout:Total download size: 26 M 2023-10-03T07:07:31.324 INFO:teuthology.orchestra.run.smithi031.stdout:Installed size: 87 M 2023-10-03T07:07:31.325 INFO:teuthology.orchestra.run.smithi031.stdout:Downloading Packages: 2023-10-03T07:07:31.370 INFO:teuthology.orchestra.run.smithi139.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 1.9 MB/s | 177 kB 00:00 2023-10-03T07:07:31.435 INFO:teuthology.orchestra.run.smithi114.stdout:(5/13): buildah-1.31.3-1.module_el8+664+4072b3a 12 MB/s | 8.8 MB 00:00 2023-10-03T07:07:31.454 INFO:teuthology.orchestra.run.smithi139.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 297 kB/s | 25 kB 00:00 2023-10-03T07:07:31.460 INFO:teuthology.orchestra.run.smithi114.stdout:(6/13): python3-podman-4.6.0-1.module_el8+582+a 818 kB/s | 177 kB 00:00 2023-10-03T07:07:31.535 INFO:teuthology.orchestra.run.smithi114.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.2 MB/s | 94 kB 00:00 2023-10-03T07:07:31.546 INFO:teuthology.orchestra.run.smithi139.stdout:(7/13): buildah-1.31.3-1.module_el8+664+4072b3a 11 MB/s | 8.8 MB 00:00 2023-10-03T07:07:31.552 INFO:teuthology.orchestra.run.smithi114.stdout:(8/13): python3-pytoml-0.1.14-5.git7dea353.el8. 269 kB/s | 25 kB 00:00 2023-10-03T07:07:31.571 INFO:teuthology.orchestra.run.smithi139.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 804 kB/s | 94 kB 00:00 2023-10-03T07:07:31.611 INFO:teuthology.orchestra.run.smithi114.stdout:(9/13): netavark-1.7.0-1.module_el8+487+8e42a27 9.1 MB/s | 3.7 MB 00:00 2023-10-03T07:07:31.638 INFO:teuthology.orchestra.run.smithi139.stdout:(9/13): netavark-1.7.0-1.module_el8+487+8e42a27 9.3 MB/s | 3.7 MB 00:00 2023-10-03T07:07:31.720 INFO:teuthology.orchestra.run.smithi114.stdout:(10/13): toolbox-0.0.99.4-5.module_el8+649+e2de 15 MB/s | 2.5 MB 00:00 2023-10-03T07:07:31.730 INFO:teuthology.orchestra.run.smithi139.stdout:(10/13): udica-0.2.6-20.module_el8+487+8e42a277 575 kB/s | 53 kB 00:00 2023-10-03T07:07:31.736 INFO:teuthology.orchestra.run.smithi114.stdout:(11/13): udica-0.2.6-20.module_el8+487+8e42a277 421 kB/s | 53 kB 00:00 2023-10-03T07:07:31.789 INFO:teuthology.orchestra.run.smithi139.stdout:(11/13): toolbox-0.0.99.4-5.module_el8+649+e2de 12 MB/s | 2.5 MB 00:00 2023-10-03T07:07:31.820 INFO:teuthology.orchestra.run.smithi114.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 409 kB/s | 41 kB 00:00 2023-10-03T07:07:31.839 INFO:teuthology.orchestra.run.smithi139.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 378 kB/s | 41 kB 00:00 2023-10-03T07:07:31.931 INFO:teuthology.orchestra.run.smithi139.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 21 MB/s | 8.1 MB 00:00 2023-10-03T07:07:31.931 INFO:teuthology.orchestra.run.smithi139.stdout:-------------------------------------------------------------------------------- 2023-10-03T07:07:31.931 INFO:teuthology.orchestra.run.smithi139.stdout:Total 18 MB/s | 26 MB 00:01 2023-10-03T07:07:31.969 INFO:teuthology.orchestra.run.smithi031.stdout:(1/13): cockpit-podman-75-1.module_el8+664+4072 1.6 MB/s | 739 kB 00:00 2023-10-03T07:07:32.002 INFO:teuthology.orchestra.run.smithi031.stdout:(2/13): aardvark-dns-1.7.0-1.module_el8+487+8e4 2.0 MB/s | 1.0 MB 00:00 2023-10-03T07:07:32.021 INFO:teuthology.orchestra.run.smithi114.stdout:(13/13): skopeo-1.13.3-1.module_el8+664+4072b3a 17 MB/s | 8.1 MB 00:00 2023-10-03T07:07:32.021 INFO:teuthology.orchestra.run.smithi114.stdout:-------------------------------------------------------------------------------- 2023-10-03T07:07:32.021 INFO:teuthology.orchestra.run.smithi114.stdout:Total 17 MB/s | 26 MB 00:01 2023-10-03T07:07:32.044 INFO:teuthology.orchestra.run.smithi031.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 505 kB/s | 38 kB 00:00 2023-10-03T07:07:32.086 INFO:teuthology.orchestra.run.smithi031.stdout:(4/13): crun-1.8.7-1.module_el8+661+d1afb926.x8 2.8 MB/s | 239 kB 00:00 2023-10-03T07:07:32.170 INFO:teuthology.orchestra.run.smithi031.stdout:(5/13): python3-podman-4.6.0-1.module_el8+582+a 2.1 MB/s | 177 kB 00:00 2023-10-03T07:07:32.194 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction check 2023-10-03T07:07:32.215 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction check succeeded. 2023-10-03T07:07:32.215 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction test 2023-10-03T07:07:32.245 INFO:teuthology.orchestra.run.smithi031.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 331 kB/s | 25 kB 00:00 2023-10-03T07:07:32.286 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction check 2023-10-03T07:07:32.307 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction check succeeded. 2023-10-03T07:07:32.307 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction test 2023-10-03T07:07:32.354 INFO:teuthology.orchestra.run.smithi031.stdout:(7/13): buildah-1.31.3-1.module_el8+664+4072b3a 11 MB/s | 8.8 MB 00:00 2023-10-03T07:07:32.404 INFO:teuthology.orchestra.run.smithi031.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 590 kB/s | 94 kB 00:00 2023-10-03T07:07:32.457 INFO:teuthology.orchestra.run.smithi139.stdout:Transaction test succeeded. 2023-10-03T07:07:32.461 INFO:teuthology.orchestra.run.smithi139.stdout:Running transaction 2023-10-03T07:07:32.471 INFO:teuthology.orchestra.run.smithi031.stdout:(9/13): netavark-1.7.0-1.module_el8+487+8e42a27 8.8 MB/s | 3.7 MB 00:00 2023-10-03T07:07:32.546 INFO:teuthology.orchestra.run.smithi031.stdout:(10/13): udica-0.2.6-20.module_el8+487+8e42a277 704 kB/s | 53 kB 00:00 2023-10-03T07:07:32.557 INFO:teuthology.orchestra.run.smithi114.stdout:Transaction test succeeded. 2023-10-03T07:07:32.561 INFO:teuthology.orchestra.run.smithi114.stdout:Running transaction 2023-10-03T07:07:32.621 INFO:teuthology.orchestra.run.smithi031.stdout:(11/13): yajl-2.1.0-12.el8.x86_64.rpm 548 kB/s | 41 kB 00:00 2023-10-03T07:07:32.713 INFO:teuthology.orchestra.run.smithi031.stdout:(12/13): skopeo-1.13.3-1.module_el8+664+4072b3a 23 MB/s | 8.1 MB 00:00 2023-10-03T07:07:32.755 INFO:teuthology.orchestra.run.smithi031.stdout:(13/13): toolbox-0.0.99.4-5.module_el8+649+e2de 7.2 MB/s | 2.5 MB 00:00 2023-10-03T07:07:32.755 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T07:07:32.756 INFO:teuthology.orchestra.run.smithi031.stdout:Total 18 MB/s | 26 MB 00:01 2023-10-03T07:07:33.044 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction check 2023-10-03T07:07:33.068 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction check succeeded. 2023-10-03T07:07:33.068 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction test 2023-10-03T07:07:33.139 INFO:teuthology.orchestra.run.smithi139.stdout: Preparing : 1/1 2023-10-03T07:07:33.261 INFO:teuthology.orchestra.run.smithi114.stdout: Preparing : 1/1 2023-10-03T07:07:33.348 INFO:teuthology.orchestra.run.smithi031.stdout:Transaction test succeeded. 2023-10-03T07:07:33.352 INFO:teuthology.orchestra.run.smithi031.stdout:Running transaction 2023-10-03T07:07:33.368 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-03T07:07:33.486 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-03T07:07:33.530 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-03T07:07:33.682 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-03T07:07:33.744 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-03T07:07:33.891 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-03T07:07:33.975 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:34.046 INFO:teuthology.orchestra.run.smithi031.stdout: Preparing : 1/1 2023-10-03T07:07:34.126 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:34.273 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-10-03T07:07:34.449 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-10-03T07:07:34.554 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-03T07:07:34.643 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-10-03T07:07:34.681 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-03T07:07:34.909 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:35.470 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 5/13 2023-10-03T07:07:35.613 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:35.804 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:35.851 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-03T07:07:36.022 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-03T07:07:36.064 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-03T07:07:36.210 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-03T07:07:36.336 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-03T07:07:36.454 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:36.467 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-03T07:07:36.718 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 7/13 2023-10-03T07:07:36.751 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-03T07:07:36.876 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-03T07:07:36.906 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 8/13 2023-10-03T07:07:37.171 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 9/13 2023-10-03T07:07:37.591 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : udica-0.2.6-20.module_el8+487+8e42a277.noarch 10/13 2023-10-03T07:07:37.759 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:37.881 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:37.981 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-03T07:07:38.121 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-03T07:07:38.154 INFO:teuthology.orchestra.run.smithi139.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:38.314 INFO:teuthology.orchestra.run.smithi114.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:38.563 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:38.573 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:38.574 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-03T07:07:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:38.750 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:38.751 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-03T07:07:38.773 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 12/13 2023-10-03T07:07:38.947 INFO:teuthology.orchestra.run.smithi031.stdout: Installing : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-03T07:07:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:39.356 INFO:teuthology.orchestra.run.smithi139.stdout:Installed: 2023-10-03T07:07:39.356 INFO:teuthology.orchestra.run.smithi139.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:39.357 INFO:teuthology.orchestra.run.smithi139.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-03T07:07:39.358 INFO:teuthology.orchestra.run.smithi139.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-03T07:07:39.358 INFO:teuthology.orchestra.run.smithi139.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-03T07:07:39.358 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:07:39.358 INFO:teuthology.orchestra.run.smithi139.stdout:Complete! 2023-10-03T07:07:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: Running scriptlet: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 13/13 2023-10-03T07:07:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86 1/13 2023-10-03T07:07:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2/13 2023-10-03T07:07:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 3/13 2023-10-03T07:07:39.466 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : crun-1.8.7-1.module_el8+661+d1afb926.x86_64 5/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 6/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-podman-4.6.0-1.module_el8+582+a2558ae2.noa 7/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 10/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 11/13 2023-10-03T07:07:39.467 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : udica-0.2.6-20.module_el8+487+8e42a277.noarch 12/13 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout:Installed: 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:39.537 INFO:teuthology.orchestra.run.smithi114.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout: 2023-10-03T07:07:39.538 INFO:teuthology.orchestra.run.smithi114.stdout:Complete! 2023-10-03T07:07:39.649 DEBUG:teuthology.parallel:result is None 2023-10-03T07:07:39.814 DEBUG:teuthology.parallel:result is None 2023-10-03T07:07:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-10-03T07:07:40.511 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout:Installed: 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: aardvark-dns-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: buildah-1:1.31.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: cockpit-podman-75-1.module_el8+664+4072b3ae.noarch 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: crun-1.8.7-1.module_el8+661+d1afb926.x86_64 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: netavark-2:1.7.0-1.module_el8+487+8e42a277.x86_64 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-podman-4.6.0-1.module_el8+582+a2558ae2.noarch 2023-10-03T07:07:40.512 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: skopeo-2:1.13.3-1.module_el8+664+4072b3ae.x86_64 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: toolbox-0.0.99.4-5.module_el8+649+e2deac1d.x86_64 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: udica-0.2.6-20.module_el8+487+8e42a277.noarch 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: yajl-2.1.0-12.el8.x86_64 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:07:40.513 INFO:teuthology.orchestra.run.smithi031.stdout:Complete! 2023-10-03T07:07:41.015 DEBUG:teuthology.parallel:result is None 2023-10-03T07:07:41.015 INFO:teuthology.run_tasks:Running task cephadm... 2023-10-03T07:07:41.085 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'bcea8c48517ff154f392584022efa0718f51c9f2'} 2023-10-03T07:07:41.085 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-03T07:07:41.086 INFO:tasks.cephadm:Cluster fsid is 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:07:41.086 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-10-03T07:07:41.086 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.31', 'mon.b': '172.21.15.114', 'mon.c': '172.21.15.139'} 2023-10-03T07:07:41.086 INFO:tasks.cephadm:First mon is mon.a on smithi031 2023-10-03T07:07:41.087 INFO:tasks.cephadm:First mgr is x 2023-10-03T07:07:41.087 INFO:tasks.cephadm:Normalizing hostnames... 2023-10-03T07:07:41.087 DEBUG:teuthology.orchestra.run.smithi031:> sudo hostname $(hostname -s) 2023-10-03T07:07:41.123 DEBUG:teuthology.orchestra.run.smithi114:> sudo hostname $(hostname -s) 2023-10-03T07:07:41.159 DEBUG:teuthology.orchestra.run.smithi139:> sudo hostname $(hostname -s) 2023-10-03T07:07:41.194 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2023-10-03T07:07:41.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-03T07:07:41.363 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': 'bcea8c48517ff154f392584022efa0718f51c9f2', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic/72782/', 'root_build_cause': 'SCMTRIGGER', 'version': '18.0.0-5817-gbcea8c48', 'node_name': '172.21.4.65+confusa03', 'job_name': 'ceph-dev-new-build/ARCH=arm64,AVAILABLE_ARCH=arm64,AVAILABLE_DIST=centos8,DIST=centos8,MACHINE_SIZE=gigantic', 'package_manager_version': '18.0.0-5817.gbcea8c48'}, 'url': 'https://2.chacra.ceph.com/r/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/flavors/default/', 'distro_codename': None, 'modified': '2023-09-25 21:06:38.915379', 'distro_version': '8', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-barakda-cephadm-nvmeof', 'chacra_url': 'https://2.chacra.ceph.com/repos/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/flavors/default/', 'archs': ['x86_64', 'source', 'arm64'], 'distro': 'centos'}] 2023-10-03T07:07:41.516 INFO:tasks.util.chacra:got chacra host 2.chacra.ceph.com, ref wip-barakda-cephadm-nvmeof, sha1 bcea8c48517ff154f392584022efa0718f51c9f2 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F8%2Fx86_64&flavor=default&sha1=bcea8c48517ff154f392584022efa0718f51c9f2 2023-10-03T07:07:41.516 INFO:tasks.cephadm:Discovered cachra url: https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm 2023-10-03T07:07:41.516 DEBUG:teuthology.orchestra.run.smithi031:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:42.177 INFO:teuthology.orchestra.run.smithi031.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Oct 3 07:07 /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:42.177 DEBUG:teuthology.orchestra.run.smithi114:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:42.585 INFO:teuthology.orchestra.run.smithi114.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Oct 3 07:07 /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:42.585 DEBUG:teuthology.orchestra.run.smithi139:> curl --silent -L https://2.chacra.ceph.com/binaries/ceph/wip-barakda-cephadm-nvmeof/bcea8c48517ff154f392584022efa0718f51c9f2/centos/8/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:43.126 INFO:teuthology.orchestra.run.smithi139.stdout:-rw-rw-r--. 1 ubuntu ubuntu 705688 Oct 3 07:07 /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:43.127 DEBUG:teuthology.orchestra.run.smithi031:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:43.152 DEBUG:teuthology.orchestra.run.smithi114:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:43.175 DEBUG:teuthology.orchestra.run.smithi139:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-10-03T07:07:43.209 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 on all hosts... 2023-10-03T07:07:43.209 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 pull 2023-10-03T07:07:43.212 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 pull 2023-10-03T07:07:43.218 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 pull 2023-10-03T07:07:44.735 INFO:teuthology.orchestra.run.smithi114.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-03T07:07:44.739 INFO:teuthology.orchestra.run.smithi031.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-03T07:07:44.764 INFO:teuthology.orchestra.run.smithi139.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-03T07:08:12.977 INFO:teuthology.orchestra.run.smithi139.stdout:{ 2023-10-03T07:08:12.978 INFO:teuthology.orchestra.run.smithi139.stdout: "ceph_version": "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)", 2023-10-03T07:08:12.978 INFO:teuthology.orchestra.run.smithi139.stdout: "image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", 2023-10-03T07:08:12.978 INFO:teuthology.orchestra.run.smithi139.stdout: "repo_digests": [ 2023-10-03T07:08:12.978 INFO:teuthology.orchestra.run.smithi139.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7" 2023-10-03T07:08:12.978 INFO:teuthology.orchestra.run.smithi139.stdout: ] 2023-10-03T07:08:12.979 INFO:teuthology.orchestra.run.smithi139.stdout:} 2023-10-03T07:08:31.524 INFO:teuthology.orchestra.run.smithi114.stdout:{ 2023-10-03T07:08:31.524 INFO:teuthology.orchestra.run.smithi114.stdout: "ceph_version": "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)", 2023-10-03T07:08:31.525 INFO:teuthology.orchestra.run.smithi114.stdout: "image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", 2023-10-03T07:08:31.525 INFO:teuthology.orchestra.run.smithi114.stdout: "repo_digests": [ 2023-10-03T07:08:31.525 INFO:teuthology.orchestra.run.smithi114.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7" 2023-10-03T07:08:31.525 INFO:teuthology.orchestra.run.smithi114.stdout: ] 2023-10-03T07:08:31.525 INFO:teuthology.orchestra.run.smithi114.stdout:} 2023-10-03T07:08:34.837 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-03T07:08:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph_version": "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)", 2023-10-03T07:08:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "image_id": "fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863", 2023-10-03T07:08:34.837 INFO:teuthology.orchestra.run.smithi031.stdout: "repo_digests": [ 2023-10-03T07:08:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:34fad0ea24358428207f7cbc4d6027320c5a86fd3de6f51aaec11e44cfc648d7" 2023-10-03T07:08:34.838 INFO:teuthology.orchestra.run.smithi031.stdout: ] 2023-10-03T07:08:34.838 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-03T07:08:34.864 DEBUG:teuthology.orchestra.run.smithi031:> sudo mkdir -p /etc/ceph 2023-10-03T07:08:34.901 DEBUG:teuthology.orchestra.run.smithi114:> sudo mkdir -p /etc/ceph 2023-10-03T07:08:34.939 DEBUG:teuthology.orchestra.run.smithi139:> sudo mkdir -p /etc/ceph 2023-10-03T07:08:34.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 777 /etc/ceph 2023-10-03T07:08:35.013 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 777 /etc/ceph 2023-10-03T07:08:35.046 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 777 /etc/ceph 2023-10-03T07:08:35.078 INFO:tasks.cephadm:Writing seed config... 2023-10-03T07:08:35.079 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-10-03T07:08:35.079 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-10-03T07:08:35.079 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-10-03T07:08:35.079 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-10-03T07:08:35.080 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-10-03T07:08:35.080 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-10-03T07:08:35.080 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-10-03T07:08:35.080 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:08:35.080 DEBUG:teuthology.orchestra.run.smithi031:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-10-03T07:08:35.104 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8ad20b58-61bb-11ee-8db5-212e2dc638e7 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2023-10-03T07:08:35.104 DEBUG:teuthology.orchestra.run.smithi031:mon.a> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service 2023-10-03T07:08:35.147 DEBUG:teuthology.orchestra.run.smithi031:mgr.x> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x.service 2023-10-03T07:08:35.182 INFO:journalctl@ceph.mon.a.smithi031.stdout:-- Logs begin at Tue 2023-10-03 06:59:00 UTC. -- 2023-10-03T07:08:35.183 INFO:tasks.cephadm:Bootstrapping... 2023-10-03T07:08:35.183 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 -v bootstrap --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.31 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:08:35.217 INFO:journalctl@ceph.mgr.x.smithi031.stdout:-- Logs begin at Tue 2023-10-03 06:59:00 UTC. -- 2023-10-03T07:08:35.374 INFO:teuthology.orchestra.run.smithi031.stdout:-------------------------------------------------------------------------------- 2023-10-03T07:08:35.374 INFO:teuthology.orchestra.run.smithi031.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2', '-v', 'bootstrap', '--fsid', '8ad20b58-61bb-11ee-8db5-212e2dc638e7', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.31', '--skip-admin-label'] 2023-10-03T07:08:35.413 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.6.1 2023-10-03T07:08:35.413 INFO:teuthology.orchestra.run.smithi031.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2023-10-03T07:08:35.414 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying podman|docker is present... 2023-10-03T07:08:35.451 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.6.1 2023-10-03T07:08:35.451 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying lvm2 is present... 2023-10-03T07:08:35.452 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying time synchronization is in place... 2023-10-03T07:08:35.459 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-03T07:08:35.459 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-03T07:08:35.466 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-03T07:08:35.466 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:08:35.473 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2023-10-03T07:08:35.479 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2023-10-03T07:08:35.480 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2023-10-03T07:08:35.480 INFO:teuthology.orchestra.run.smithi031.stdout:Repeating the final host check... 2023-10-03T07:08:35.511 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout 4.6.1 2023-10-03T07:08:35.511 INFO:teuthology.orchestra.run.smithi031.stdout:podman (/bin/podman) version 4.6.1 is present 2023-10-03T07:08:35.512 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl is present 2023-10-03T07:08:35.512 INFO:teuthology.orchestra.run.smithi031.stdout:lvcreate is present 2023-10-03T07:08:35.518 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2023-10-03T07:08:35.519 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2023-10-03T07:08:35.525 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2023-10-03T07:08:35.525 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:08:35.531 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout enabled 2023-10-03T07:08:35.538 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout active 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Unit chronyd.service is enabled and running 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Host looks OK 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Cluster fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Acquiring lock 140393565048224 on /run/cephadm/8ad20b58-61bb-11ee-8db5-212e2dc638e7.lock 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Lock 140393565048224 acquired on /run/cephadm/8ad20b58-61bb-11ee-8db5-212e2dc638e7.lock 2023-10-03T07:08:35.539 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 3300 ... 2023-10-03T07:08:35.540 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying IP 172.21.15.31 port 6789 ... 2023-10-03T07:08:35.540 INFO:teuthology.orchestra.run.smithi031.stdout:Base mon IP(s) is [172.21.15.31:3300, 172.21.15.31:6789], mon addrv is [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2023-10-03T07:08:35.543 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.31 metric 100 2023-10-03T07:08:35.543 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.31 metric 100 2023-10-03T07:08:35.547 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2023-10-03T07:08:35.547 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-10-03T07:08:35.547 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1578sec hoplimit 64 pref medium 2023-10-03T07:08:35.550 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-10-03T07:08:35.550 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:caba/64 scope link 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2023-10-03T07:08:35.551 INFO:teuthology.orchestra.run.smithi031.stdout:Mon IP `172.21.15.31` is in CIDR network `172.21.0.0/20` 2023-10-03T07:08:35.552 INFO:teuthology.orchestra.run.smithi031.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-10-03T07:08:35.552 INFO:teuthology.orchestra.run.smithi031.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-10-03T07:08:35.553 INFO:teuthology.orchestra.run.smithi031.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-03T07:08:35.901 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stdout fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863 2023-10-03T07:08:35.901 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2... 2023-10-03T07:08:35.901 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Getting image source signatures 2023-10-03T07:08:35.901 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:556fc22cf1e739705d2013195476c145689a6e17c5a9071e0d49fdc6e036642a 2023-10-03T07:08:35.902 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying blob sha256:3f4fd5c3c800c6da9ef94691df203a05a5bbbf32c483d09d620ec06eef42ea2b 2023-10-03T07:08:35.902 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Copying config sha256:fa922c4449069cae14b17e9ffaef6cf64d1d7080cea061a20dc1d8f318e65863 2023-10-03T07:08:35.902 INFO:teuthology.orchestra.run.smithi031.stdout:/bin/podman: stderr Writing manifest to image destination 2023-10-03T07:08:36.522 INFO:teuthology.orchestra.run.smithi031.stdout:ceph: stdout ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev) 2023-10-03T07:08:36.522 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph version: ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev) 2023-10-03T07:08:36.522 INFO:teuthology.orchestra.run.smithi031.stdout:Extracting ceph user uid/gid from container image... 2023-10-03T07:08:37.061 INFO:teuthology.orchestra.run.smithi031.stdout:stat: stdout 167 167 2023-10-03T07:08:37.062 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial keys... 2023-10-03T07:08:37.595 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQD1vRtlLkJnDhAAh/A0Rmi0v1aufN4MQdTUrA== 2023-10-03T07:08:38.115 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQD1vRtlPcr2LRAAMNZhf5NSYWGsvsxjCRVOgg== 2023-10-03T07:08:38.656 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph-authtool: stdout AQD2vRtlX77MERAAW8C6sK/LO46pmMT/4M/EGA== 2023-10-03T07:08:38.656 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial monmap... 2023-10-03T07:08:39.207 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = pacific 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:monmaptool for a [v2:172.21.15.31:3300,v1:172.21.15.31:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:setting min_mon_release = pacific 2023-10-03T07:08:39.208 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: set fsid to 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:39.209 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-10-03T07:08:39.209 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:08:39.209 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mon... 2023-10-03T07:08:39.793 INFO:teuthology.orchestra.run.smithi031.stdout:create mon.a on 2023-10-03T07:08:40.033 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-10-03T07:08:40.166 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target → /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target. 2023-10-03T07:08:40.166 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target → /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target. 2023-10-03T07:08:40.357 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a 2023-10-03T07:08:40.357 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service: Unit ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service not loaded. 2023-10-03T07:08:40.463 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target.wants/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service → /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@.service. 2023-10-03T07:08:40.860 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-03T07:08:40.860 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-10-03T07:08:40.869 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-03T07:08:40.869 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:08:40.869 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-10-03T07:08:40.870 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2023-10-03T07:08:40.870 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon to start... 2023-10-03T07:08:40.870 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mon... 2023-10-03T07:08:41.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:40 smithi031 ceph-mon[106786]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout cluster: 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout id: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout services: 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.422033s) 2023-10-03T07:08:41.641 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout data: 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2023-10-03T07:08:41.642 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout pgs: 2023-10-03T07:08:41.643 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:41.643 INFO:teuthology.orchestra.run.smithi031.stdout:mon is available 2023-10-03T07:08:41.643 INFO:teuthology.orchestra.run.smithi031.stdout:Assimilating anything we can from ceph.conf... 2023-10-03T07:08:42.122 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-03T07:08:42.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} 2023-10-03T07:08:42.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: fsmap 2023-10-03T07:08:42.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: osdmap e1: 0 total, 0 up, 0 in 2023-10-03T07:08:42.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: mgrmap e1: no daemons active 2023-10-03T07:08:42.123 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:41 smithi031 ceph-mon[106786]: from='client.? 172.21.15.31:0/1817391281' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-03T07:08:42.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:42.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2023-10-03T07:08:42.479 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.31:3300,v1:172.21.15.31:6789] 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2023-10-03T07:08:42.480 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-03T07:08:42.481 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-03T07:08:42.481 INFO:teuthology.orchestra.run.smithi031.stdout:Generating new minimal ceph.conf... 2023-10-03T07:08:43.148 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:42 smithi031 ceph-mon[106786]: from='client.? 172.21.15.31:0/1441620292' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-03T07:08:43.148 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:42 smithi031 ceph-mon[106786]: from='client.? 172.21.15.31:0/1441620292' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-10-03T07:08:43.149 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:42 smithi031 ceph-mon[106786]: from='client.? 172.21.15.31:0/2455895181' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:08:43.212 INFO:teuthology.orchestra.run.smithi031.stdout:Restarting the monitor... 2023-10-03T07:08:43.410 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 systemd[1]: Stopping Ceph mon.a for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:08:43.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a[106782]: 2023-10-03T07:08:43.408+0000 7fe295166700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:08:43.726 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a[106782]: 2023-10-03T07:08:43.408+0000 7fe295166700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2023-10-03T07:08:43.978 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 bash[107041]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a 2023-10-03T07:08:44.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 systemd[1]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service: Succeeded. 2023-10-03T07:08:44.256 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 systemd[1]: Stopped Ceph mon.a for 8ad20b58-61bb-11ee-8db5-212e2dc638e7. 2023-10-03T07:08:44.257 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:43 smithi031 systemd[1]: Starting Ceph mon.a for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:08:44.331 INFO:teuthology.orchestra.run.smithi031.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 bash[107151]: 1b0e77539eb07822dbcca61631eca273c87daa19bd9defa1db119190d8910a72 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: set uid:gid to 167:167 (ceph:ceph) 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev), process ceph-mon, pid 2 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: pidfile_write: ignore empty --pid-file 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 systemd[1]: Started Ceph mon.a for 8ad20b58-61bb-11ee-8db5-212e2dc638e7. 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: load: jerasure load: lrc 2023-10-03T07:08:44.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: RocksDB version: 7.9.2 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Git sha 0 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Compile date 2023-09-25 19:01:54 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: DB SUMMARY 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: DB Session ID: V1DU7WJ4P19HCIK1AGDJ 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: CURRENT file: CURRENT 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: IDENTITY file: IDENTITY 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2023-10-03T07:08:44.732 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 82822 ; 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.error_if_exists: 0 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.create_if_missing: 0 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.paranoid_checks: 1 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.env: 0x563ee5107da0 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.fs: PosixFileSystem 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.info_log: 0x563ee7507460 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_file_opening_threads: 16 2023-10-03T07:08:44.733 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.statistics: (nil) 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.use_fsync: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_log_file_size: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.keep_log_file_num: 1000 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.recycle_log_file_num: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_fallocate: 1 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_mmap_reads: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_mmap_writes: 0 2023-10-03T07:08:44.734 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.use_direct_reads: 0 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.create_missing_column_families: 0 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.db_log_dir: 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.wal_dir: 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-03T07:08:44.735 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.advise_random_on_open: 1 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.db_write_buffer_size: 0 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.write_buffer_manager: 0x563ee74c14a0 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.rate_limiter: (nil) 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.wal_recovery_mode: 2 2023-10-03T07:08:44.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enable_thread_tracking: 0 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enable_pipelined_write: 0 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.unordered_write: 0 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.row_cache: None 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.wal_filter: None 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_ingest_behind: 0 2023-10-03T07:08:44.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.two_write_queues: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.manual_wal_flush: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.wal_compression: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.atomic_flush: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.log_readahead_size: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.best_efforts_recovery: 0 2023-10-03T07:08:44.738 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.allow_data_in_errors: 0 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.db_host_id: __hostname__ 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_background_jobs: 2 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_background_compactions: -1 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_subcompactions: 1 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-03T07:08:44.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_total_wal_size: 0 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_open_files: -1 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bytes_per_sync: 0 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-03T07:08:44.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_readahead_size: 0 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_background_flushes: -1 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Compression algorithms supported: 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kZSTD supported: 0 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kXpressCompression supported: 0 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kLZ4HCCompression supported: 1 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kZlibCompression supported: 1 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kSnappyCompression supported: 1 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kLZ4Compression supported: 1 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: kBZip2Compression supported: 0 2023-10-03T07:08:44.741 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.merge_operator: 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_filter: None 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_filter_factory: None 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.sst_partitioner_factory: None 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-03T07:08:44.742 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x563ee7507560) 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks: 1 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: pin_top_level_index_and_filter: 1 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_type: 0 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_index_type: 0 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_shortening: 1 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: checksum: 4 2023-10-03T07:08:44.743 INFO:journalctl@ceph.mon.a.smithi031.stdout: no_block_cache: 0 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache: 0x563ee755f1f0 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_name: BinnedLRUCache 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_options: 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: capacity : 536870912 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_shard_bits : 4 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: strict_capacity_limit : 0 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: high_pri_pool_ratio: 0.000 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_cache_compressed: (nil) 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: persistent_cache: (nil) 2023-10-03T07:08:44.744 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size: 4096 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_size_deviation: 10 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_restart_interval: 16 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: index_block_restart_interval: 1 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: metadata_block_size: 4096 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: partition_filters: 0 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: use_delta_encoding: 1 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: filter_policy: bloomfilter 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: whole_key_filtering: 1 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: verify_compression: 0 2023-10-03T07:08:44.745 INFO:journalctl@ceph.mon.a.smithi031.stdout: read_amp_bytes_per_bit: 0 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: format_version: 5 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_index_compression: 1 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: block_align: 0 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: max_auto_readahead_size: 262144 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: prepopulate_block_cache: 0 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: initial_auto_readahead_size: 8192 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout: num_file_reads_for_auto_readahead: 2 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.write_buffer_size: 33554432 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_write_buffer_number: 2 2023-10-03T07:08:44.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression: NoCompression 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression: Disabled 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.prefix_extractor: nullptr 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.num_levels: 7 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-03T07:08:44.747 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.level: 32767 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.strategy: 0 2023-10-03T07:08:44.748 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.enabled: false 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-03T07:08:44.749 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.target_file_size_base: 67108864 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-03T07:08:44.750 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.arena_block_size: 1048576 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.disable_auto_compactions: 0 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-03T07:08:44.751 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.table_properties_collectors: 2023-10-03T07:08:44.752 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.inplace_update_support: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.bloom_locality: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.max_successive_merges: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.paranoid_file_checks: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.force_consistency_checks: 1 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.report_bg_io_stats: 0 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.ttl: 2592000 2023-10-03T07:08:44.753 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enable_blob_files: false 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.min_blob_size: 0 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_file_size: 268435456 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-03T07:08:44.754 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.blob_file_starting_level: 0 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 8143507b-7577-4a69-8326-b142f3dbfdfd 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316924334940, "job": 1, "event": "recovery_started", "wal_files": [9]} 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316924336403, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 79747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 242, "table_properties": {"data_size": 77927, "index_size": 230, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 581, "raw_key_size": 9724, "raw_average_key_size": 47, "raw_value_size": 72370, "raw_average_value_size": 353, "num_data_blocks": 10, "num_entries": 205, "num_filter_entries": 205, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696316924, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "8143507b-7577-4a69-8326-b142f3dbfdfd", "db_session_id": "V1DU7WJ4P19HCIK1AGDJ", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316924336494, "job": 1, "event": "recovery_finished"} 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2023-10-03T07:08:44.755 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x563ee75fc000 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: DB pointer 0x563ee75e8000 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** DB Stats ** 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:08:44.756 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: L0 2/0 79.71 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: Sum 2/0 79.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 78.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.757 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** Compaction Stats [default] ** 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 78.5 0.00 0.00 1 0.001 0 0 0.0 0.0 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-03T07:08:44.758 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: Cumulative compaction: 0.00 GB write, 18.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: Interval compaction: 0.00 GB write, 18.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache BinnedLRUCache@0x563ee755f1f0#2 capacity: 512.00 MB usage: 1.19 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.3e-05 secs_since: 0 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.77 KB,0.000146031%) IndexBlock(2,0.42 KB,8.04663e-05%) Misc(1,0.00 KB,0%) 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: starting mon.a rank 0 at public addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] at bind addrs [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???) e1 preinit fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:44.759 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).mds e1 new map 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).mds e1 print_map 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: e1 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: legacy client fscid: -1 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout: No filesystems configured 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:08:44.760 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mon.a is new leader, mons a in quorum (ranks 0) 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: monmap e1: 1 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0]} removed_ranks: {} 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: fsmap 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: osdmap e1: 0 total, 0 up, 0 in 2023-10-03T07:08:44.761 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:44 smithi031 ceph-mon[107170]: mgrmap e1: no daemons active 2023-10-03T07:08:45.046 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote config to /etc/ceph/ceph.conf 2023-10-03T07:08:45.079 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:08:45.079 INFO:teuthology.orchestra.run.smithi031.stdout:Creating mgr... 2023-10-03T07:08:45.080 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:9283 ... 2023-10-03T07:08:45.080 INFO:teuthology.orchestra.run.smithi031.stdout:Verifying port 0.0.0.0:8765 ... 2023-10-03T07:08:45.304 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x 2023-10-03T07:08:45.304 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x.service: Unit ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x.service not loaded. 2023-10-03T07:08:45.402 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7.target.wants/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x.service → /etc/systemd/system/ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@.service. 2023-10-03T07:08:45.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:45 smithi031 systemd[1]: Starting Ceph mgr.x for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:08:45.768 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-03T07:08:45.768 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-10-03T07:08:45.779 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-03T07:08:45.779 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:08:45.779 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-10-03T07:08:45.779 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to enable service . firewalld.service is not available 2023-10-03T07:08:45.788 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-03T07:08:45.788 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-10-03T07:08:45.797 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-03T07:08:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:08:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-10-03T07:08:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2023-10-03T07:08:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr to start... 2023-10-03T07:08:45.798 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr... 2023-10-03T07:08:46.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:45 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3859757959' entity='client.admin' 2023-10-03T07:08:46.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:45 smithi031 bash[107378]: 2e68f5c773076cb0936f53e47ea5152e53eaa0a29d1ce83a27da56bdc96ee39c 2023-10-03T07:08:46.068 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:45 smithi031 systemd[1]: Started Ceph mgr.x for 8ad20b58-61bb-11ee-8db5-212e2dc638e7. 2023-10-03T07:08:46.068 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:45 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:45.905+0000 7f2e66e67200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T07:08:46.393 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:46 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:46.107+0000 7f2e66e67200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T07:08:46.556 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "8ad20b58-61bb-11ee-8db5-212e2dc638e7", 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-03T07:08:46.557 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 1, 2023-10-03T07:08:46.558 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-03T07:08:46.559 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-03T07:08:46.560 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-03T07:08:46.563 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-03T07:08:46.563 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-03T07:08:46.564 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-03T07:08:46.565 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-10-03T07:08:46.566 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-10-03T07:08:40.883713+0000", 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:08:46.567 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (1/15)... 2023-10-03T07:08:46.706 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:46 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:46.488+0000 7f2e66e67200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T07:08:47.065 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:46 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3033846392' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-03T07:08:47.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:47 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:47.064+0000 7f2e66e67200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-03T07:08:47.793 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:47 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:47.319+0000 7f2e66e67200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-03T07:08:48.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:47 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:47.793+0000 7f2e66e67200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T07:08:48.980 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:48 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:48.605+0000 7f2e66e67200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T07:08:48.980 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:48 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:48.706+0000 7f2e66e67200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "8ad20b58-61bb-11ee-8db5-212e2dc638e7", 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-03T07:08:49.306 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:49.307 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-03T07:08:49.308 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-03T07:08:49.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-03T07:08:49.309 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-03T07:08:49.313 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-03T07:08:49.314 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-03T07:08:49.315 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-10-03T07:08:40.883713+0000", 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:49.316 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:49.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-03T07:08:49.317 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:08:49.317 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (2/15)... 2023-10-03T07:08:49.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1896017630' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-03T07:08:49.816 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:49.406+0000 7f2e66e67200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T07:08:49.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:49.501+0000 7f2e66e67200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T07:08:49.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:49.600+0000 7f2e66e67200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T07:08:50.306 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:49.858+0000 7f2e66e67200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T07:08:50.306 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:49 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:49.961+0000 7f2e66e67200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T07:08:50.306 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:50 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:50.066+0000 7f2e66e67200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T07:08:50.566 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:50 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:50.305+0000 7f2e66e67200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T07:08:50.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:50 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:50.414+0000 7f2e66e67200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T07:08:51.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:50 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:50.619+0000 7f2e66e67200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T07:08:51.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:50 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:50.715+0000 7f2e66e67200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T07:08:51.689 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:51 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:51.592+0000 7f2e66e67200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T07:08:52.021 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:52.021 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:08:52.021 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "8ad20b58-61bb-11ee-8db5-212e2dc638e7", 2023-10-03T07:08:52.021 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:52.022 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-03T07:08:52.023 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:52.024 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:51 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3182798581' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-03T07:08:52.024 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:51 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:51.816+0000 7f2e66e67200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T07:08:52.027 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-03T07:08:52.028 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-03T07:08:52.029 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": false, 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-03T07:08:52.030 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-10-03T07:08:40.883713+0000", 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:08:52.031 INFO:teuthology.orchestra.run.smithi031.stdout:mgr not available, waiting (3/15)... 2023-10-03T07:08:52.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:52 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:52.021+0000 7f2e66e67200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:08:53.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:52 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:52.773+0000 7f2e66e67200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T07:08:53.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:52 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:52.977+0000 7f2e66e67200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:53.183+0000 7f2e66e67200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:53.315+0000 7f2e66e67200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: Activating manager daemon x 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: mgrmap e2: x(active, starting, since 0.0035867s) 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-03T07:08:53.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: Manager daemon x is now available 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' 2023-10-03T07:08:53.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:53 smithi031 ceph-mon[107170]: from='mgr.14100 172.21.15.31:0/3646745160' entity='mgr.x' 2023-10-03T07:08:54.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:54.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:08:54.979 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsid": "8ad20b58-61bb-11ee-8db5-212e2dc638e7", 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "health": { 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "checks": {}, 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mutes": [] 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.980 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum": [ 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 0 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "a" 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:54.981 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2023-10-03T07:08:54.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "monmap": { 2023-10-03T07:08:54.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:54.982 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "reef", 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_mons": 1 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osdmap": { 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2023-10-03T07:08:54.986 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgmap": { 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2023-10-03T07:08:54.987 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "fsmap": { 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "by_rank": [], 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "up:standby": 0 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap": { 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modules": [ 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "iostat", 2023-10-03T07:08:54.988 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "nfs", 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "restful" 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ], 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "servicemap": { 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 1, 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "modified": "2023-10-03T07:08:40.883713+0000", 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "services": {} 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout }, 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "progress_events": {} 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:08:54.989 INFO:teuthology.orchestra.run.smithi031.stdout:mgr is available 2023-10-03T07:08:55.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:55 smithi031 ceph-mon[107170]: mgrmap e3: x(active, since 1.0073s) 2023-10-03T07:08:55.772 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:55 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/2489492846' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-10-03T07:08:55.845 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:55.845 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [global] 2023-10-03T07:08:55.845 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout fsid = 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [mgr] 2023-10-03T07:08:55.846 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2023-10-03T07:08:55.847 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 2023-10-03T07:08:55.847 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout [osd] 2023-10-03T07:08:55.847 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2023-10-03T07:08:55.847 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2023-10-03T07:08:55.847 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling cephadm module... 2023-10-03T07:08:56.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:56 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1068819830' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-10-03T07:08:57.637 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-mon[107170]: mgrmap e4: x(active, since 3s) 2023-10-03T07:08:57.637 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/544975505' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-10-03T07:08:57.638 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: ignoring --setuser ceph since I am not root 2023-10-03T07:08:57.638 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: ignoring --setgroup ceph since I am not root 2023-10-03T07:08:57.638 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:57.542+0000 7f8b8d11f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T07:08:58.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:57 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:57.741+0000 7f8b8d11f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T07:08:58.496 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:58.120+0000 7f8b8d11f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T07:08:58.496 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/544975505' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-10-03T07:08:58.497 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-mon[107170]: mgrmap e5: x(active, since 4s) 2023-10-03T07:08:58.497 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1233634917' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-03T07:08:58.635 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 5, 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2023-10-03T07:08:58.636 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 5... 2023-10-03T07:08:58.748 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:58.707+0000 7f8b8d11f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-03T07:08:59.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:58 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:58.969+0000 7f8b8d11f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-03T07:08:59.816 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:08:59 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:08:59.445+0000 7f8b8d11f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T07:09:00.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:00 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:00.250+0000 7f8b8d11f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T07:09:00.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:00 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:00.348+0000 7f8b8d11f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T07:09:01.317 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.045+0000 7f8b8d11f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T07:09:01.317 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.141+0000 7f8b8d11f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T07:09:01.317 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.241+0000 7f8b8d11f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T07:09:01.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.500+0000 7f8b8d11f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T07:09:01.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.604+0000 7f8b8d11f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T07:09:01.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.710+0000 7f8b8d11f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T07:09:02.268 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:01 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:01.951+0000 7f8b8d11f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T07:09:02.268 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:02 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:02.060+0000 7f8b8d11f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T07:09:02.268 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:02 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:02.268+0000 7f8b8d11f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T07:09:02.566 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:02 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:02.365+0000 7f8b8d11f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T07:09:03.566 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:03 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:03.227+0000 7f8b8d11f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T07:09:03.567 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:03 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:03.447+0000 7f8b8d11f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T07:09:04.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:03 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:03.648+0000 7f8b8d11f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:09:04.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:04 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:04.398+0000 7f8b8d11f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T07:09:04.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:04 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:04.602+0000 7f8b8d11f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:09:04.808 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:04 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:04.807+0000 7f8b8d11f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T07:09:05.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:04 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:04.937+0000 7f8b8d11f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: Active manager daemon x restarted 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: Activating manager daemon x 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: osdmap e2: 0 total, 0 up, 0 in 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: mgrmap e6: x(active, starting, since 0.00366732s) 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-03T07:09:05.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: Manager daemon x is now available 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-03T07:09:05.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:05 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-03T07:09:06.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:06 smithi031 ceph-mon[107170]: Found migration_current of "None". Setting to last migration. 2023-10-03T07:09:06.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:06 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:06.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:06 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:06.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:06 smithi031 ceph-mon[107170]: mgrmap e7: x(active, since 1.00856s) 2023-10-03T07:09:06.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:06 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:07.638 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:05] ENGINE Bus STARTING 2023-10-03T07:09:07.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-03T07:09:07.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:05] ENGINE Serving on http://172.21.15.31:8765 2023-10-03T07:09:07.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-03T07:09:07.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:06] ENGINE Serving on https://172.21.15.31:7150 2023-10-03T07:09:07.639 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:07 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:06] ENGINE Bus STARTED 2023-10-03T07:09:08.718 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:09:08.719 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2023-10-03T07:09:08.719 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-03T07:09:08.719 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:09:08.719 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 5 is available 2023-10-03T07:09:08.719 INFO:teuthology.orchestra.run.smithi031.stdout:Setting orchestrator backend to cephadm... 2023-10-03T07:09:08.721 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:08 smithi031 ceph-mon[107170]: mgrmap e8: x(active, since 2s) 2023-10-03T07:09:10.650 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout value unchanged 2023-10-03T07:09:10.650 INFO:teuthology.orchestra.run.smithi031.stdout:Generating ssh key... 2023-10-03T07:09:10.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:10 smithi031 ceph-mon[107170]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:10.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:10 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:10.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:10 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:10.756 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:10 smithi031 ceph-mon[107170]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:11.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-mon[107170]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:11.731 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-mon[107170]: Generating ssh key... 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: Generating public/private rsa key pair. 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: Your identification has been saved in /tmp/tmpik6poqql/key. 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: Your public key has been saved in /tmp/tmpik6poqql/key.pub. 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: The key fingerprint is: 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: SHA256:XqNhQvlhkU/PLROJeV0N0tZeu3erK8RjkvbWz7XQXrA ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: The key's randomart image is: 2023-10-03T07:09:11.732 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: +---[RSA 3072]----+ 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | .. o.+.+o| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | ...+ +.+ +| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | o oo + + .o| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | . o .. = ...| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | . S = o. .| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | + B * .+o| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | + = o.E.*| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | + .+.+| 2023-10-03T07:09:11.733 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: | . .o+= | 2023-10-03T07:09:11.734 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:11 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: +----[SHA256]-----+ 2023-10-03T07:09:12.799 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:12.800 INFO:teuthology.orchestra.run.smithi031.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-10-03T07:09:12.800 INFO:teuthology.orchestra.run.smithi031.stdout:Adding key to root@localhost authorized_keys... 2023-10-03T07:09:12.801 INFO:teuthology.orchestra.run.smithi031.stdout:Adding host smithi031... 2023-10-03T07:09:12.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:12 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:12.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:12 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:13.601 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:13 smithi031 ceph-mon[107170]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:13.601 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:13 smithi031 ceph-mon[107170]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi031", "addr": "172.21.15.31", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:15.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:15 smithi031 ceph-mon[107170]: Deploying cephadm binary to smithi031 2023-10-03T07:09:16.001 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Added host 'smithi031' with addr '172.21.15.31' 2023-10-03T07:09:16.001 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mon service... 2023-10-03T07:09:16.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:16 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:16.860 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:16 smithi031 ceph-mon[107170]: Added host smithi031 2023-10-03T07:09:16.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:16 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:16.861 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:16 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:17.183 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2023-10-03T07:09:17.184 INFO:teuthology.orchestra.run.smithi031.stdout:Deploying unmanaged mgr service... 2023-10-03T07:09:18.064 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2023-10-03T07:09:18.811 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:18 smithi031 ceph-mon[107170]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:18.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:18 smithi031 ceph-mon[107170]: Saving service mon spec with placement count:5 2023-10-03T07:09:18.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:18 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:18.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:18 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:18.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:18 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:19.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:19 smithi031 ceph-mon[107170]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:19.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:19 smithi031 ceph-mon[107170]: Saving service mgr spec with placement count:2 2023-10-03T07:09:19.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:19 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:19.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:19 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1563331033' entity='client.admin' 2023-10-03T07:09:19.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:19 smithi031 ceph-mon[107170]: from='mgr.14120 172.21.15.31:0/3498728335' entity='mgr.x' 2023-10-03T07:09:20.348 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling the dashboard module... 2023-10-03T07:09:21.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:20 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/837013628' entity='client.admin' 2023-10-03T07:09:22.046 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:21 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3714019784' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-10-03T07:09:22.046 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:21 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: ignoring --setuser ceph since I am not root 2023-10-03T07:09:22.046 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:21 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: ignoring --setgroup ceph since I am not root 2023-10-03T07:09:22.046 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:22.015+0000 7f3230064200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2023-10-03T07:09:22.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:22.213+0000 7f3230064200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2023-10-03T07:09:23.006 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:22.600+0000 7f3230064200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2023-10-03T07:09:23.006 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3714019784' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-10-03T07:09:23.007 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-mon[107170]: mgrmap e9: x(active, since 16s) 2023-10-03T07:09:23.007 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:22 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1184277461' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-10-03T07:09:23.174 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "epoch": 9, 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "available": true, 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "active_name": "x", 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "num_standby": 0 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for the mgr to restart... 2023-10-03T07:09:23.175 INFO:teuthology.orchestra.run.smithi031.stdout:Waiting for mgr epoch 9... 2023-10-03T07:09:23.295 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:23 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:23.172+0000 7f3230064200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2023-10-03T07:09:23.566 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:23 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:23.436+0000 7f3230064200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2023-10-03T07:09:24.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:23 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:23.914+0000 7f3230064200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2023-10-03T07:09:25.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:24 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:24.715+0000 7f3230064200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2023-10-03T07:09:25.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:24 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:24.814+0000 7f3230064200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2023-10-03T07:09:25.816 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:25 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:25.507+0000 7f3230064200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2023-10-03T07:09:25.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:25 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:25.605+0000 7f3230064200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2023-10-03T07:09:25.817 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:25 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:25.706+0000 7f3230064200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2023-10-03T07:09:26.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:25 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:25.964+0000 7f3230064200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2023-10-03T07:09:26.316 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.068+0000 7f3230064200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2023-10-03T07:09:26.317 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.175+0000 7f3230064200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2023-10-03T07:09:26.738 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.418+0000 7f3230064200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2023-10-03T07:09:26.738 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.527+0000 7f3230064200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2023-10-03T07:09:27.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.738+0000 7f3230064200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2023-10-03T07:09:27.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:26 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:26.835+0000 7f3230064200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2023-10-03T07:09:28.066 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:27 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:27.720+0000 7f3230064200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2023-10-03T07:09:28.067 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:27 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:27.938+0000 7f3230064200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2023-10-03T07:09:28.566 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:28 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:28.138+0000 7f3230064200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:09:29.301 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:28 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:28.891+0000 7f3230064200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2023-10-03T07:09:29.301 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:29 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:29.094+0000 7f3230064200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2023-10-03T07:09:29.558 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:29 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:29.301+0000 7f3230064200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2023-10-03T07:09:29.558 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:09:29 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:09:29.432+0000 7f3230064200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2023-10-03T07:09:30.490 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: Active manager daemon x restarted 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: Activating manager daemon x 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: osdmap e3: 0 total, 0 up, 0 in 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: mgrmap e10: x(active, starting, since 0.00357626s) 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-10-03T07:09:30.491 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: Manager daemon x is now available 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2023-10-03T07:09:30.492 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2023-10-03T07:09:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout { 2023-10-03T07:09:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2023-10-03T07:09:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout "initialized": true 2023-10-03T07:09:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout } 2023-10-03T07:09:30.807 INFO:teuthology.orchestra.run.smithi031.stdout:mgr epoch 9 is available 2023-10-03T07:09:30.808 INFO:teuthology.orchestra.run.smithi031.stdout:Generating a dashboard self-signed certificate... 2023-10-03T07:09:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:29] ENGINE Bus STARTING 2023-10-03T07:09:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:30] ENGINE Serving on http://172.21.15.31:8765 2023-10-03T07:09:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:30] ENGINE Serving on https://172.21.15.31:7150 2023-10-03T07:09:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: [03/Oct/2023:07:09:30] ENGINE Bus STARTED 2023-10-03T07:09:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:31.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-10-03T07:09:31.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: mgrmap e11: x(active, since 1.00903s) 2023-10-03T07:09:31.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-10-03T07:09:31.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:32.163 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2023-10-03T07:09:32.164 INFO:teuthology.orchestra.run.smithi031.stdout:Creating initial admin user... 2023-10-03T07:09:32.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:32 smithi031 ceph-mon[107170]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:32.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:32 smithi031 ceph-mon[107170]: mgrmap e12: x(active, since 2s) 2023-10-03T07:09:33.338 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$jEXUAdYVe6000Q/Ou0uhTe127xWtyKrluJRnE6Km7WyZ.pRxX3Vkq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1696316972, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-10-03T07:09:33.339 INFO:teuthology.orchestra.run.smithi031.stdout:Fetching dashboard port number... 2023-10-03T07:09:33.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:33 smithi031 ceph-mon[107170]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:33.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:33 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:34.229 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stdout 8443 2023-10-03T07:09:34.241 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2023-10-03T07:09:34.241 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout disabled 2023-10-03T07:09:34.250 INFO:teuthology.orchestra.run.smithi031.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2023-10-03T07:09:34.250 INFO:teuthology.orchestra.run.smithi031.stdout:systemctl: stdout inactive 2023-10-03T07:09:34.251 INFO:teuthology.orchestra.run.smithi031.stdout:firewalld.service is not enabled 2023-10-03T07:09:34.251 INFO:teuthology.orchestra.run.smithi031.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2023-10-03T07:09:34.252 INFO:teuthology.orchestra.run.smithi031.stdout:Ceph Dashboard is now available at: 2023-10-03T07:09:34.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:34.253 INFO:teuthology.orchestra.run.smithi031.stdout: URL: https://smithi031.front.sepia.ceph.com:8443/ 2023-10-03T07:09:34.253 INFO:teuthology.orchestra.run.smithi031.stdout: User: admin 2023-10-03T07:09:34.253 INFO:teuthology.orchestra.run.smithi031.stdout: Password: amn9jcpubu 2023-10-03T07:09:34.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:Saving cluster configuration to /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config directory 2023-10-03T07:09:34.254 INFO:teuthology.orchestra.run.smithi031.stdout:Enabling autotune for osd_memory_target 2023-10-03T07:09:34.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:34 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/38963413' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-10-03T07:09:36.138 INFO:teuthology.orchestra.run.smithi031.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2023-10-03T07:09:36.138 INFO:teuthology.orchestra.run.smithi031.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout:Or, if you are only running a single cluster on this host: 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2023-10-03T07:09:36.139 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:Please consider enabling telemetry to help improve Ceph: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: ceph telemetry on 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout:For more information see: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.140 INFO:teuthology.orchestra.run.smithi031.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2023-10-03T07:09:36.141 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:36.141 INFO:teuthology.orchestra.run.smithi031.stdout:Bootstrap complete. 2023-10-03T07:09:36.186 INFO:tasks.cephadm:Fetching config... 2023-10-03T07:09:36.186 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:09:36.186 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-10-03T07:09:36.208 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-10-03T07:09:36.208 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:09:36.208 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-10-03T07:09:36.269 INFO:tasks.cephadm:Fetching mon keyring... 2023-10-03T07:09:36.269 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:09:36.270 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/keyring of=/dev/stdout 2023-10-03T07:09:36.347 INFO:tasks.cephadm:Fetching pub ssh key... 2023-10-03T07:09:36.347 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:09:36.347 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-10-03T07:09:36.409 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-10-03T07:09:36.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-03T07:09:36.491 INFO:teuthology.orchestra.run.smithi031.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:36.504 DEBUG:teuthology.orchestra.run.smithi114:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-03T07:09:36.550 INFO:teuthology.orchestra.run.smithi114.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:36.561 DEBUG:teuthology.orchestra.run.smithi139:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-10-03T07:09:36.607 INFO:teuthology.orchestra.run.smithi139.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDp4WlhAOrXonz9L61p5g0ZH8lBogbN45wbiHqVxMi6HxDfIdUY7uTlYldf/BsNaLrNP+u9/O19bNnc5XTnVe3pmra3XGhWJQUx87snzbD9dRMv3nhmrS+VI4dEPuAQfx1XPfzZBGz2TVTCIFJzevudAS6sHo5ZIDsfKQFvg3arWarQCODWmj7P3bo/9pjGbgEHeoD8GRK2nSLHJ+IlF4ncgjq9VE4kgpiWI+ZKWiJOmyYXEAD03tM1NKYsHPaH7jiKTexrppaxTl8C0g2q2U4UZq+4D8WDqMBoNMsgQJJPeL5orABzVmPq3EoeMIyIwJu6ZYIDLCabcPm/BbNKTO4NhRoSDQwdHq139yP8qh8foYoR2M4IaPUl7uNp+S81lydO1VyNuhTEi3W6NOMtdfTqAOmH7ekBEiGoJHz/bSDBBGi8nZwN7aveRgPgfQeXUd14GQrrHh/NZrjqEnhesAqvzGeEpGB64pKVQChW9gLVK093MnLFVBglMz7pXtditrU= ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:36.620 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-10-03T07:09:36.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:36 smithi031 ceph-mon[107170]: mgrmap e13: x(active, since 6s) 2023-10-03T07:09:36.648 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:36 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1379898092' entity='client.admin' 2023-10-03T07:09:36.872 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:38.780 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-10-03T07:09:38.780 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-10-03T07:09:39.070 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:39.523 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/27444838' entity='client.admin' 2023-10-03T07:09:39.523 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:39.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:39.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi031", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:09:39.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:39.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:39.524 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:40.781 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi114 2023-10-03T07:09:40.781 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:09:40.781 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.conf 2023-10-03T07:09:40.804 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:09:40.804 DEBUG:teuthology.orchestra.run.smithi114:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:40.865 INFO:tasks.cephadm:Adding host smithi114 to orchestrator... 2023-10-03T07:09:40.865 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch host add smithi114 2023-10-03T07:09:41.123 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: Updating smithi031:/etc/ceph/ceph.conf 2023-10-03T07:09:41.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:41 smithi031 ceph-mon[107170]: Updating smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:09:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:42 smithi031 ceph-mon[107170]: Updating smithi031:/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:42 smithi031 ceph-mon[107170]: Updating smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-03T07:09:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:42 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:42 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:42 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:44.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:43 smithi031 ceph-mon[107170]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi114", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:44.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:43 smithi031 ceph-mon[107170]: Deploying cephadm binary to smithi114 2023-10-03T07:09:44.541 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi114' with addr '172.21.15.114' 2023-10-03T07:09:44.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-03T07:09:45.156 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:45 smithi031 ceph-mon[107170]: Added host smithi114 2023-10-03T07:09:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:46.342 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:46.342 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}] 2023-10-03T07:09:46.729 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi139 2023-10-03T07:09:46.729 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:09:46.729 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.conf 2023-10-03T07:09:46.752 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:09:46.752 DEBUG:teuthology.orchestra.run.smithi139:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:46.814 INFO:tasks.cephadm:Adding host smithi139 to orchestrator... 2023-10-03T07:09:46.814 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch host add smithi139 2023-10-03T07:09:47.065 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:47.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:46 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:47.195 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:46 smithi031 ceph-mon[107170]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:09:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:49.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi114", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:09:49.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:09:49.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:09:49.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:48 smithi031 ceph-mon[107170]: Updating smithi114:/etc/ceph/ceph.conf 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: Updating smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi139", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: Updating smithi114:/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: Updating smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: Deploying cephadm binary to smithi139 2023-10-03T07:09:50.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:50.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:50.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:49 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:50.534 INFO:teuthology.orchestra.run.smithi031.stdout:Added host 'smithi139' with addr '172.21.15.139' 2023-10-03T07:09:50.907 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch host ls --format=json 2023-10-03T07:09:51.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:50 smithi031 ceph-mon[107170]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:51.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:50 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:51.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:50 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:51.154 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:52.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:51 smithi031 ceph-mon[107170]: Added host smithi139 2023-10-03T07:09:52.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:51 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:52.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:51 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:52.237 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:09:52.237 INFO:teuthology.orchestra.run.smithi031.stdout:[{"addr": "172.21.15.31", "hostname": "smithi031", "labels": [], "status": ""}, {"addr": "172.21.15.114", "hostname": "smithi114", "labels": [], "status": ""}, {"addr": "172.21.15.139", "hostname": "smithi139", "labels": [], "status": ""}] 2023-10-03T07:09:52.611 INFO:tasks.cephadm:Setting crush tunables to default 2023-10-03T07:09:52.611 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd crush tunables default 2023-10-03T07:09:52.860 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:09:53.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:52 smithi031 ceph-mon[107170]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:53.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:52 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi139", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:09:54.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:09:54.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:53 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:09:54.872 INFO:teuthology.orchestra.run.smithi031.stderr:adjusted tunables profile to default 2023-10-03T07:09:55.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:55.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: Updating smithi139:/etc/ceph/ceph.conf 2023-10-03T07:09:55.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3054118926' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-10-03T07:09:55.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:55.137 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:55.138 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:55.209 INFO:tasks.cephadm:Adding mon.a on smithi031 2023-10-03T07:09:55.210 INFO:tasks.cephadm:Adding mon.b on smithi114 2023-10-03T07:09:55.210 INFO:tasks.cephadm:Adding mon.c on smithi139 2023-10-03T07:09:55.210 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch apply mon '3;smithi031:172.21.15.31=a;smithi114:172.21.15.114=b;smithi139:172.21.15.139=c' 2023-10-03T07:09:55.440 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:09:56.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:55 smithi031 ceph-mon[107170]: Updating smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:09:56.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:55 smithi031 ceph-mon[107170]: Updating smithi139:/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:09:56.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:55 smithi031 ceph-mon[107170]: Updating smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.client.admin.keyring 2023-10-03T07:09:56.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:55 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3054118926' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-10-03T07:09:56.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:55 smithi031 ceph-mon[107170]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:09:56.474 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mon update... 2023-10-03T07:09:56.843 DEBUG:teuthology.orchestra.run.smithi114:mon.b> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.b.service 2023-10-03T07:09:56.846 DEBUG:teuthology.orchestra.run.smithi139:mon.c> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.c.service 2023-10-03T07:09:56.849 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-03T07:09:56.849 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph mon dump -f json 2023-10-03T07:09:56.879 INFO:journalctl@ceph.mon.c.smithi139.stdout:-- Logs begin at Tue 2023-10-03 06:59:09 UTC. -- 2023-10-03T07:09:56.882 INFO:journalctl@ceph.mon.b.smithi114.stdout:-- Logs begin at Tue 2023-10-03 06:59:12 UTC. -- 2023-10-03T07:09:57.289 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:09:57.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:09:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:09:58.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:57 smithi031 ceph-mon[107170]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi031:172.21.15.31=a;smithi114:172.21.15.114=b;smithi139:172.21.15.139=c", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:09:58.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:57 smithi031 ceph-mon[107170]: Saving service mon spec with placement smithi031:172.21.15.31=a;smithi114:172.21.15.114=b;smithi139:172.21.15.139=c;count:3 2023-10-03T07:09:58.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:57 smithi031 ceph-mon[107170]: Deploying daemon mon.c on smithi139 2023-10-03T07:09:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:58 smithi139 systemd[1]: Starting Ceph mon.c for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:09:58.896 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:09:58.896 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":1,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","modified":"2023-10-03T07:08:38.851451Z","created":"2023-10-03T07:08:38.851451Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-10-03T07:09:58.898 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 1 2023-10-03T07:09:59.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:09:58 smithi031 ceph-mon[107170]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:59.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: set uid:gid to 167:167 (ceph:ceph) 2023-10-03T07:09:59.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev), process ceph-mon, pid 2 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: pidfile_write: ignore empty --pid-file 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 bash[109260]: 4fb98de11a3542e3d1badc3db1e67a09efca49221bf2297a61c83d95d8dd6800 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: load: jerasure load: lrc 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: RocksDB version: 7.9.2 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Git sha 0 2023-10-03T07:09:59.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Compile date 2023-09-25 19:01:54 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: DB SUMMARY 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: DB Session ID: OG60QB7QEBMPJWX0NZUH 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: CURRENT file: CURRENT 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: IDENTITY file: IDENTITY 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2023-10-03T07:09:59.375 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.error_if_exists: 0 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.create_if_missing: 0 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.paranoid_checks: 1 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.flush_verify_memtable_count: 1 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.env: 0x557f5d833da0 2023-10-03T07:09:59.376 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.fs: PosixFileSystem 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.info_log: 0x557f5e901460 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_file_opening_threads: 16 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.statistics: (nil) 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.use_fsync: 0 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_log_file_size: 0 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_manifest_file_size: 1073741824 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.log_file_time_to_roll: 0 2023-10-03T07:09:59.377 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.keep_log_file_num: 1000 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.recycle_log_file_num: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_fallocate: 1 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_mmap_reads: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_mmap_writes: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.use_direct_reads: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.create_missing_column_families: 0 2023-10-03T07:09:59.378 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.db_log_dir: 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.wal_dir: 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.table_cache_numshardbits: 6 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.WAL_ttl_seconds: 0 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.WAL_size_limit_MB: 0 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.manifest_preallocation_size: 4194304 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.is_fd_close_on_exec: 1 2023-10-03T07:09:59.379 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.advise_random_on_open: 1 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.db_write_buffer_size: 0 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.write_buffer_manager: 0x557f5e8bb4a0 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.access_hint_on_compaction_start: 1 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.random_access_max_buffer_size: 1048576 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.use_adaptive_mutex: 0 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.rate_limiter: (nil) 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-10-03T07:09:59.380 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.wal_recovery_mode: 2 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enable_thread_tracking: 0 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enable_pipelined_write: 0 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.unordered_write: 0 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_concurrent_memtable_write: 1 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.write_thread_max_yield_usec: 100 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.write_thread_slow_yield_usec: 3 2023-10-03T07:09:59.381 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.row_cache: None 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.wal_filter: None 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.avoid_flush_during_recovery: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_ingest_behind: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.two_write_queues: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.manual_wal_flush: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.wal_compression: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.atomic_flush: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-10-03T07:09:59.382 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.persist_stats_to_disk: 0 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.write_dbid_to_manifest: 0 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.log_readahead_size: 0 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.file_checksum_gen_factory: Unknown 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.best_efforts_recovery: 0 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.allow_data_in_errors: 0 2023-10-03T07:09:59.383 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.db_host_id: __hostname__ 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enforce_single_del_contracts: true 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_background_jobs: 2 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_background_compactions: -1 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_subcompactions: 1 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.avoid_flush_during_shutdown: 0 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.delayed_write_rate : 16777216 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_total_wal_size: 0 2023-10-03T07:09:59.384 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.stats_dump_period_sec: 600 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.stats_persist_period_sec: 600 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.stats_history_buffer_size: 1048576 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_open_files: -1 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bytes_per_sync: 0 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.wal_bytes_per_sync: 0 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.strict_bytes_per_sync: 0 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_readahead_size: 0 2023-10-03T07:09:59.385 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_background_flushes: -1 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Compression algorithms supported: 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kZSTDNotFinalCompression supported: 0 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kZSTD supported: 0 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kXpressCompression supported: 0 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kLZ4HCCompression supported: 1 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kZlibCompression supported: 1 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kSnappyCompression supported: 1 2023-10-03T07:09:59.386 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kLZ4Compression supported: 1 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: kBZip2Compression supported: 0 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Fast CRC32 supported: Supported on x86 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: DMutex implementation: pthread_mutex_t 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.merge_operator: 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_filter: None 2023-10-03T07:09:59.387 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_filter_factory: None 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.sst_partitioner_factory: None 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.memtable_factory: SkipListFactory 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.table_factory: BlockBasedTable 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x557f5e901560) 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks: 1 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout: pin_top_level_index_and_filter: 1 2023-10-03T07:09:59.388 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_type: 0 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_index_type: 0 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_shortening: 1 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: data_block_hash_table_util_ratio: 0.750000 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: checksum: 4 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: no_block_cache: 0 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache: 0x557f5e9591f0 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_name: BinnedLRUCache 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_options: 2023-10-03T07:09:59.389 INFO:journalctl@ceph.mon.c.smithi139.stdout: capacity : 536870912 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_shard_bits : 4 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: strict_capacity_limit : 0 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: high_pri_pool_ratio: 0.000 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_cache_compressed: (nil) 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: persistent_cache: (nil) 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size: 4096 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_size_deviation: 10 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_restart_interval: 16 2023-10-03T07:09:59.390 INFO:journalctl@ceph.mon.c.smithi139.stdout: index_block_restart_interval: 1 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: metadata_block_size: 4096 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: partition_filters: 0 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: use_delta_encoding: 1 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: filter_policy: bloomfilter 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: whole_key_filtering: 1 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: verify_compression: 0 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: read_amp_bytes_per_bit: 0 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: format_version: 5 2023-10-03T07:09:59.391 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_index_compression: 1 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout: block_align: 0 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout: max_auto_readahead_size: 262144 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout: prepopulate_block_cache: 0 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout: initial_auto_readahead_size: 8192 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout: num_file_reads_for_auto_readahead: 2 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.write_buffer_size: 33554432 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_write_buffer_number: 2 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression: NoCompression 2023-10-03T07:09:59.392 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression: Disabled 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.prefix_extractor: nullptr 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.num_levels: 7 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.level: 32767 2023-10-03T07:09:59.393 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.enabled: false 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.window_bits: -14 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.level: 32767 2023-10-03T07:09:59.394 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.strategy: 0 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.parallel_threads: 1 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.enabled: false 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-10-03T07:09:59.395 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.level0_stop_writes_trigger: 36 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.target_file_size_base: 67108864 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.target_file_size_multiplier: 1 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_base: 268435456 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-10-03T07:09:59.396 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_compaction_bytes: 1677721600 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.arena_block_size: 1048576 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-10-03T07:09:59.397 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.disable_auto_compactions: 0 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 systemd[1]: Started Ceph mon.c for 8ad20b58-61bb-11ee-8db5-212e2dc638e7. 2023-10-03T07:09:59.398 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.table_properties_collectors: 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.inplace_update_support: 0 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.inplace_update_num_locks: 10000 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-10-03T07:09:59.399 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.memtable_whole_key_filtering: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.memtable_huge_page_size: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.bloom_locality: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.max_successive_merges: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.optimize_filters_for_hits: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.paranoid_file_checks: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.force_consistency_checks: 1 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.report_bg_io_stats: 0 2023-10-03T07:09:59.400 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.ttl: 2592000 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.periodic_compaction_seconds: 0 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.preclude_last_level_data_seconds: 0 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.preserve_internal_time_seconds: 0 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enable_blob_files: false 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.min_blob_size: 0 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_file_size: 268435456 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_compression_type: NoCompression 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.enable_blob_garbage_collection: false 2023-10-03T07:09:59.401 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_compaction_readahead_size: 0 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.blob_file_starting_level: 0 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: ff965a9c-6986-49b7-a145-c858a6b22df2 2023-10-03T07:09:59.402 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316999140849, "job": 1, "event": "recovery_started", "wal_files": [4]} 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316999141368, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696316999, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "ff965a9c-6986-49b7-a145-c858a6b22df2", "db_session_id": "OG60QB7QEBMPJWX0NZUH", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696316999141439, "job": 1, "event": "recovery_finished"} 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x557f5e9f6000 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: DB pointer 0x557f5e9e2000 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-03T07:09:59.403 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** DB Stats ** 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:09:59.404 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** Compaction Stats [default] ** 2023-10-03T07:09:59.405 INFO:journalctl@ceph.mon.c.smithi139.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.1 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-03T07:09:59.406 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: Cumulative compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: Interval compaction: 0.00 GB write, 0.56 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache BinnedLRUCache@0x557f5e9591f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-03T07:09:59.407 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c does not exist in monmap, will attempt to join an existing cluster 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: using public_addr v2:172.21.15.139:0/0 -> [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: starting mon.c rank -1 at public addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] at bind addrs [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(???) e0 preinit fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).mds e1 new map 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).mds e1 print_map 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout: e1 2023-10-03T07:09:59.408 INFO:journalctl@ceph.mon.c.smithi139.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout: legacy client fscid: -1 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout: 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout: No filesystems configured 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-03T07:09:59.409 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: mon.c@-1(synchronizing) e1 handle_conf_change mon_allow_pool_delete 2023-10-03T07:09:59.410 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expand map: {default=false} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta from 'false' to 'false' 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expanded map: {default=false} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expand map: {default=info} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta from 'info' to 'info' 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expanded map: {default=info} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expand map: {default=daemon} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expanded map: {default=daemon} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expand map: {default=debug} 2023-10-03T07:09:59.411 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta from 'debug' to 'debug' 2023-10-03T07:09:59.412 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:09:59 smithi139 ceph-mon[109290]: expand_channel_meta expanded map: {default=debug} 2023-10-03T07:10:00.545 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-03T07:10:00.545 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph mon dump -f json 2023-10-03T07:10:00.778 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:10:04.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: mon.a calling monitor election 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: mon.c calling monitor election 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.569 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: fsmap 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: mgrmap e13: x(active, since 34s) 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: overall HEALTH_OK 2023-10-03T07:10:04.570 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:04.571 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:04.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:04.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: mon.a calling monitor election 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: mon.c calling monitor election 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:04.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: fsmap 2023-10-03T07:10:04.625 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:04.625 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: mgrmap e13: x(active, since 34s) 2023-10-03T07:10:04.625 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: overall HEALTH_OK 2023-10-03T07:10:04.625 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:04.625 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:04.907 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:10:04.907 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":2,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","modified":"2023-10-03T07:09:59.165508Z","created":"2023-10-03T07:08:38.851451Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2023-10-03T07:10:04.909 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 2 2023-10-03T07:10:05.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:05.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:05.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:05.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: Deploying daemon mon.b on smithi114 2023-10-03T07:10:05.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/1136586706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:05.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:05 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:05.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:05.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:05.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:05.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: Deploying daemon mon.b on smithi114 2023-10-03T07:10:05.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/1136586706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:05.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:05 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:06.301 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2023-10-03T07:10:06.302 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph mon dump -f json 2023-10-03T07:10:06.531 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:10:06.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:06 smithi031 ceph-mon[107170]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:06.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:06 smithi139 ceph-mon[109290]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 systemd[1]: Started Ceph mon.b for 8ad20b58-61bb-11ee-8db5-212e2dc638e7. 2023-10-03T07:10:07.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696317006772933, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1696317006, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e2bf6014-054b-4ca0-bb28-986a30b92011", "db_session_id": "ZOWF7MIM4ABZH3A66GJ6", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2023-10-03T07:10:07.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: EVENT_LOG_v1 {"time_micros": 1696317006773030, "job": 1, "event": "recovery_finished"} 2023-10-03T07:10:07.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2023-10-03T07:10:07.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2023-10-03T07:10:07.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55b658db8000 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: DB pointer 0x55b658da4000 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: rocksdb: [db/db_impl/db_impl.cc:1111] 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** DB Stats ** 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:10:07.246 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-10-03T07:10:07.247 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:10:07.248 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.249 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** Compaction Stats [default] ** 2023-10-03T07:10:07.249 INFO:journalctl@ceph.mon.b.smithi114.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2023-10-03T07:10:07.249 INFO:journalctl@ceph.mon.b.smithi114.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-10-03T07:10:07.249 INFO:journalctl@ceph.mon.b.smithi114.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.6 0.00 0.00 1 0.000 0 0 0.0 0.0 2023-10-03T07:10:07.249 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: Uptime(secs): 0.0 total, 0.0 interval 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: Flush(GB): cumulative 0.000, interval 0.000 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Total Files): cumulative 0, interval 0 2023-10-03T07:10:07.250 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(L0 Files): cumulative 0, interval 0 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: AddFile(Keys): cumulative 0, interval 0 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: Cumulative compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: Interval compaction: 0.00 GB write, 0.64 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache BinnedLRUCache@0x55b658d1b1f0#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2023-10-03T07:10:07.251 INFO:journalctl@ceph.mon.b.smithi114.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout: ** File Read Latency Histogram By Level [default] ** 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b does not exist in monmap, will attempt to join an existing cluster 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: using public_addr v2:172.21.15.114:0/0 -> [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: starting mon.b rank -1 at public addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] at bind addrs [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(???) e0 preinit fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:10:07.252 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).mds e1 new map 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).mds e1 print_map 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout: e1 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout: enable_multiple, ever_enabled_multiple: 1,1 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout: legacy client fscid: -1 2023-10-03T07:10:07.253 INFO:journalctl@ceph.mon.b.smithi114.stdout: 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout: No filesystems configured 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2023-10-03T07:10:07.254 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.255 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.a calling monitor election 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.256 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.c calling monitor election 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2023-10-03T07:10:07.257 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: monmap e2: 2 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: fsmap 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mgrmap e13: x(active, since 34s) 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: overall HEALTH_OK 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:07.258 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: Deploying daemon mon.b on smithi114 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/1136586706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:07.259 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expand map: {default=false} 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta from 'false' to 'false' 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expanded map: {default=false} 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expand map: {default=info} 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta from 'info' to 'info' 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expanded map: {default=info} 2023-10-03T07:10:07.260 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expand map: {default=daemon} 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta from 'daemon' to 'daemon' 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expanded map: {default=daemon} 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expand map: {default=debug} 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta from 'debug' to 'debug' 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: expand_channel_meta expanded map: {default=debug} 2023-10-03T07:10:07.261 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:06 smithi114 ceph-mon[109024]: mon.b@-1(synchronizing) e2 handle_conf_change mon_allow_pool_delete 2023-10-03T07:10:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: mon.a calling monitor election 2023-10-03T07:10:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: mon.c calling monitor election 2023-10-03T07:10:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: mon.b calling monitor election 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.245 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: fsmap 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:12.246 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: mgrmap e13: x(active, since 42s) 2023-10-03T07:10:12.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: overall HEALTH_OK 2023-10-03T07:10:12.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:12.247 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:12.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:12.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: mon.a calling monitor election 2023-10-03T07:10:12.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: mon.c calling monitor election 2023-10-03T07:10:12.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: mon.b calling monitor election 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: fsmap 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:12.319 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: mgrmap e13: x(active, since 42s) 2023-10-03T07:10:12.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: overall HEALTH_OK 2023-10-03T07:10:12.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:12.320 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:12.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: mon.a calling monitor election 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: mon.c calling monitor election 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: mon.b calling monitor election 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:12.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: monmap e3: 3 mons at {a=[v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0],b=[v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0],c=[v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0]} removed_ranks: {} 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: fsmap 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: osdmap e4: 0 total, 0 up, 0 in 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: mgrmap e13: x(active, since 42s) 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: overall HEALTH_OK 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:12.374 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: Updating smithi031:/etc/ceph/ceph.conf 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: Updating smithi114:/etc/ceph/ceph.conf 2023-10-03T07:10:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:12 smithi114 ceph-mon[109024]: Updating smithi139:/etc/ceph/ceph.conf 2023-10-03T07:10:13.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: Updating smithi031:/etc/ceph/ceph.conf 2023-10-03T07:10:13.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: Updating smithi114:/etc/ceph/ceph.conf 2023-10-03T07:10:13.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:12 smithi031 ceph-mon[107170]: Updating smithi139:/etc/ceph/ceph.conf 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: Updating smithi031:/etc/ceph/ceph.conf 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: Updating smithi114:/etc/ceph/ceph.conf 2023-10-03T07:10:13.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:12 smithi139 ceph-mon[109290]: Updating smithi139:/etc/ceph/ceph.conf 2023-10-03T07:10:13.660 INFO:teuthology.orchestra.run.smithi139.stdout: 2023-10-03T07:10:13.660 INFO:teuthology.orchestra.run.smithi139.stdout:{"epoch":3,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","modified":"2023-10-03T07:10:06.799941Z","created":"2023-10-03T07:08:38.851451Z","min_mon_release":18,"min_mon_release_name":"reef","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:3300","nonce":0},{"type":"v1","addr":"172.21.15.31:6789","nonce":0}]},"addr":"172.21.15.31:6789/0","public_addr":"172.21.15.31:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:3300","nonce":0},{"type":"v1","addr":"172.21.15.139:6789","nonce":0}]},"addr":"172.21.15.139:6789/0","public_addr":"172.21.15.139:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:3300","nonce":0},{"type":"v1","addr":"172.21.15.114:6789","nonce":0}]},"addr":"172.21.15.114:6789/0","public_addr":"172.21.15.114:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2023-10-03T07:10:13.662 INFO:teuthology.orchestra.run.smithi139.stderr:dumped monmap epoch 3 2023-10-03T07:10:14.024 INFO:tasks.cephadm:Generating final ceph.conf file... 2023-10-03T07:10:14.024 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph config generate-minimal-conf 2023-10-03T07:10:14.476 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: Updating smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.476 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: Updating smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: Updating smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.477 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/2133775972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: Reconfiguring mon.a (unknown last config time)... 2023-10-03T07:10:14.478 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:14.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:14.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:14.479 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:14 smithi031 ceph-mon[107170]: Reconfiguring daemon mon.a on smithi031 2023-10-03T07:10:14.505 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:10:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: Updating smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: Updating smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: Updating smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/2133775972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: Reconfiguring mon.a (unknown last config time)... 2023-10-03T07:10:14.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:14.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:14.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:14.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:14 smithi114 ceph-mon[109024]: Reconfiguring daemon mon.a on smithi031 2023-10-03T07:10:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: Updating smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: Updating smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: Updating smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/config/ceph.conf 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/2133775972' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-10-03T07:10:14.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: Reconfiguring mon.a (unknown last config time)... 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:14.874 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:14 smithi139 ceph-mon[109290]: Reconfiguring daemon mon.a on smithi031 2023-10-03T07:10:15.628 INFO:teuthology.orchestra.run.smithi031.stdout:# minimal ceph.conf for 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:10:15.628 INFO:teuthology.orchestra.run.smithi031.stdout:[global] 2023-10-03T07:10:15.628 INFO:teuthology.orchestra.run.smithi031.stdout: fsid = 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:10:15.628 INFO:teuthology.orchestra.run.smithi031.stdout: mon_host = [v2:172.21.15.31:3300/0,v1:172.21.15.31:6789/0] [v2:172.21.15.114:3300/0,v1:172.21.15.114:6789/0] [v2:172.21.15.139:3300/0,v1:172.21.15.139:6789/0] 2023-10-03T07:10:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: Reconfiguring mon.b (monmap changed)... 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: Reconfiguring daemon mon.b on smithi114 2023-10-03T07:10:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: Reconfiguring mon.b (monmap changed)... 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:15.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: Reconfiguring daemon mon.b on smithi114 2023-10-03T07:10:15.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.866 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.866 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.866 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: Reconfiguring mon.b (monmap changed)... 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: Reconfiguring daemon mon.b on smithi114 2023-10-03T07:10:15.867 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:15.999 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2023-10-03T07:10:16.000 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:10:16.000 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.conf 2023-10-03T07:10:16.039 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:10:16.040 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:10:16.116 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:10:16.116 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.conf 2023-10-03T07:10:16.152 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:10:16.152 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:10:16.222 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:10:16.223 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.conf 2023-10-03T07:10:16.253 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:10:16.253 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2023-10-03T07:10:16.323 INFO:tasks.cephadm:Adding mgr.x on smithi031 2023-10-03T07:10:16.324 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch apply mgr '1;smithi031=x' 2023-10-03T07:10:16.574 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:10:16.785 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:16.785 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.785 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: Reconfiguring mon.c (monmap changed)... 2023-10-03T07:10:16.785 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:16.785 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: Reconfiguring daemon mon.c on smithi139 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/666314393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:16.786 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: Reconfiguring mon.c (monmap changed)... 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: Reconfiguring daemon mon.c on smithi139 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/666314393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:16.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:16.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:16.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: Reconfiguring mon.c (monmap changed)... 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: Reconfiguring daemon mon.c on smithi139 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/666314393' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:16.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:17.652 INFO:teuthology.orchestra.run.smithi139.stdout:Scheduled mgr update... 2023-10-03T07:10:18.031 INFO:tasks.cephadm:Deploying OSDs... 2023-10-03T07:10:18.032 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:10:18.032 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T07:10:18.052 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T07:10:18.052 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_1 2023-10-03T07:10:18.112 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T07:10:18.112 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 390788 Links: 1 2023-10-03T07:10:18.112 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 07:09:38.048970088 +0000 2023-10-03T07:10:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 07:06:45.063728270 +0000 2023-10-03T07:10:18.113 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 07:06:45.063728270 +0000 2023-10-03T07:10:18.113 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T07:10:18.113 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T07:10:18.184 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T07:10:18.184 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T07:10:18.185 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000242597 s, 2.1 MB/s 2023-10-03T07:10:18.186 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T07:10:18.249 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_2 2023-10-03T07:10:18.311 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 390911 Links: 1 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 07:09:38.048970088 +0000 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 07:06:45.641719245 +0000 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 07:06:45.641719245 +0000 2023-10-03T07:10:18.312 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T07:10:18.312 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T07:10:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T07:10:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T07:10:18.385 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000226613 s, 2.3 MB/s 2023-10-03T07:10:18.386 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T07:10:18.449 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_3 2023-10-03T07:10:18.512 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T07:10:18.512 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:18.512 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 393723 Links: 1 2023-10-03T07:10:18.512 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:18.513 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:18.513 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 07:09:38.048970088 +0000 2023-10-03T07:10:18.513 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 07:06:46.220710203 +0000 2023-10-03T07:10:18.513 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 07:06:46.220710203 +0000 2023-10-03T07:10:18.513 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T07:10:18.513 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T07:10:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T07:10:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T07:10:18.587 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.000234943 s, 2.2 MB/s 2023-10-03T07:10:18.589 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T07:10:18.655 DEBUG:teuthology.orchestra.run.smithi031:> stat /dev/vg_nvme/lv_4 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Device: 6h/6d Inode: 396353 Links: 1 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Access: 2023-10-03 07:09:38.048970088 +0000 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Modify: 2023-10-03 07:06:46.777701506 +0000 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout:Change: 2023-10-03 07:06:46.777701506 +0000 2023-10-03T07:10:18.719 INFO:teuthology.orchestra.run.smithi031.stdout: Birth: - 2023-10-03T07:10:18.719 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T07:10:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records in 2023-10-03T07:10:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:1+0 records out 2023-10-03T07:10:18.792 INFO:teuthology.orchestra.run.smithi031.stderr:512 bytes copied, 0.0002263 s, 2.3 MB/s 2023-10-03T07:10:18.793 DEBUG:teuthology.orchestra.run.smithi031:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T07:10:18.856 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:10:18.856 DEBUG:teuthology.orchestra.run.smithi114:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T07:10:18.879 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T07:10:18.880 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_1 2023-10-03T07:10:18.940 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 384754 Links: 1 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-10-03 07:10:06.345665449 +0000 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-10-03 07:06:00.538469623 +0000 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-10-03 07:06:00.538469623 +0000 2023-10-03T07:10:18.941 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-10-03T07:10:18.941 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi031=x", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: Saving service mgr spec with placement smithi031=x;count:1 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-03T07:10:18.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:19.010 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-10-03T07:10:19.010 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-10-03T07:10:19.011 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00025631 s, 2.0 MB/s 2023-10-03T07:10:19.012 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T07:10:19.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:19.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi031=x", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: Saving service mgr spec with placement smithi031=x;count:1 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-03T07:10:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:19.073 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_2 2023-10-03T07:10:19.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:19.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "1;smithi031=x", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:19.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: Saving service mgr spec with placement smithi031=x;count:1 2023-10-03T07:10:19.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2023-10-03T07:10:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 388688 Links: 1 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-10-03 07:10:06.345665449 +0000 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-10-03 07:06:01.024459201 +0000 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-10-03 07:06:01.024459201 +0000 2023-10-03T07:10:19.133 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-10-03T07:10:19.134 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T07:10:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-10-03T07:10:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-10-03T07:10:19.201 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000230819 s, 2.2 MB/s 2023-10-03T07:10:19.202 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T07:10:19.263 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_3 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 384796 Links: 1 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-10-03 07:10:06.345665449 +0000 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-10-03 07:06:01.514448693 +0000 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-10-03 07:06:01.514448693 +0000 2023-10-03T07:10:19.322 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-10-03T07:10:19.323 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T07:10:19.389 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-10-03T07:10:19.389 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-10-03T07:10:19.389 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.000212326 s, 2.4 MB/s 2023-10-03T07:10:19.390 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T07:10:19.450 DEBUG:teuthology.orchestra.run.smithi114:> stat /dev/vg_nvme/lv_4 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout:Device: 6h/6d Inode: 388879 Links: 1 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:19.509 INFO:teuthology.orchestra.run.smithi114.stdout:Access: 2023-10-03 07:10:06.346665427 +0000 2023-10-03T07:10:19.510 INFO:teuthology.orchestra.run.smithi114.stdout:Modify: 2023-10-03 07:06:02.012438014 +0000 2023-10-03T07:10:19.510 INFO:teuthology.orchestra.run.smithi114.stdout:Change: 2023-10-03 07:06:02.012438014 +0000 2023-10-03T07:10:19.510 INFO:teuthology.orchestra.run.smithi114.stdout: Birth: - 2023-10-03T07:10:19.510 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T07:10:19.576 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records in 2023-10-03T07:10:19.576 INFO:teuthology.orchestra.run.smithi114.stderr:1+0 records out 2023-10-03T07:10:19.576 INFO:teuthology.orchestra.run.smithi114.stderr:512 bytes copied, 0.00022151 s, 2.3 MB/s 2023-10-03T07:10:19.577 DEBUG:teuthology.orchestra.run.smithi114:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T07:10:19.636 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:10:19.636 DEBUG:teuthology.orchestra.run.smithi139:> dd if=/scratch_devs of=/dev/stdout 2023-10-03T07:10:19.659 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2023-10-03T07:10:19.659 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_1 2023-10-03T07:10:19.720 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 388813 Links: 1 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-10-03 07:09:58.795146839 +0000 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-10-03 07:06:00.530593483 +0000 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-10-03 07:06:00.530593483 +0000 2023-10-03T07:10:19.721 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-10-03T07:10:19.721 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2023-10-03T07:10:19.792 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-10-03T07:10:19.913 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-10-03T07:10:19.914 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.00030741 s, 1.7 MB/s 2023-10-03T07:10:19.915 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2023-10-03T07:10:19.978 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_2 2023-10-03T07:10:20.039 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 388877 Links: 1 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-10-03 07:09:58.796146818 +0000 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-10-03 07:06:01.032583143 +0000 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-10-03 07:06:01.032583143 +0000 2023-10-03T07:10:20.040 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-10-03T07:10:20.040 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2023-10-03T07:10:20.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:19 smithi031 ceph-mon[107170]: Reconfiguring mgr.x (unknown last config time)... 2023-10-03T07:10:20.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:19 smithi031 ceph-mon[107170]: Reconfiguring daemon mgr.x on smithi031 2023-10-03T07:10:20.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.108 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:19 smithi139 ceph-mon[109290]: Reconfiguring mgr.x (unknown last config time)... 2023-10-03T07:10:20.109 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:19 smithi139 ceph-mon[109290]: Reconfiguring daemon mgr.x on smithi031 2023-10-03T07:10:20.109 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.109 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.111 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-10-03T07:10:20.111 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-10-03T07:10:20.111 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000236655 s, 2.2 MB/s 2023-10-03T07:10:20.112 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2023-10-03T07:10:20.175 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_3 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 386906 Links: 1 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:20.236 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-10-03 07:09:58.796146818 +0000 2023-10-03T07:10:20.237 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-10-03 07:06:01.531572864 +0000 2023-10-03T07:10:20.237 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-10-03 07:06:01.531572864 +0000 2023-10-03T07:10:20.237 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-10-03T07:10:20.237 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2023-10-03T07:10:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:19 smithi114 ceph-mon[109024]: Reconfiguring mgr.x (unknown last config time)... 2023-10-03T07:10:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:19 smithi114 ceph-mon[109024]: Reconfiguring daemon mgr.x on smithi031 2023-10-03T07:10:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:20.306 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-10-03T07:10:20.306 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-10-03T07:10:20.306 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000255851 s, 2.0 MB/s 2023-10-03T07:10:20.307 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2023-10-03T07:10:20.368 DEBUG:teuthology.orchestra.run.smithi139:> stat /dev/vg_nvme/lv_4 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout:Device: 6h/6d Inode: 395412 Links: 1 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout:Context: system_u:object_r:device_t:s0 2023-10-03T07:10:20.428 INFO:teuthology.orchestra.run.smithi139.stdout:Access: 2023-10-03 07:09:58.796146818 +0000 2023-10-03T07:10:20.429 INFO:teuthology.orchestra.run.smithi139.stdout:Modify: 2023-10-03 07:06:02.043562318 +0000 2023-10-03T07:10:20.429 INFO:teuthology.orchestra.run.smithi139.stdout:Change: 2023-10-03 07:06:02.043562318 +0000 2023-10-03T07:10:20.429 INFO:teuthology.orchestra.run.smithi139.stdout: Birth: - 2023-10-03T07:10:20.429 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2023-10-03T07:10:20.497 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records in 2023-10-03T07:10:20.497 INFO:teuthology.orchestra.run.smithi139.stderr:1+0 records out 2023-10-03T07:10:20.497 INFO:teuthology.orchestra.run.smithi139.stderr:512 bytes copied, 0.000241569 s, 2.1 MB/s 2023-10-03T07:10:20.498 DEBUG:teuthology.orchestra.run.smithi139:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2023-10-03T07:10:20.558 INFO:tasks.cephadm:Deploying osd.0 on smithi031 with /dev/vg_nvme/lv_4... 2023-10-03T07:10:20.558 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-03T07:10:20.806 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:10:20.926 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:20 smithi031 ceph-mon[107170]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:21.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:20 smithi139 ceph-mon[109290]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:21.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:20 smithi114 ceph-mon[109024]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:22.073 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-03T07:10:22.073 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-03T07:10:22.073 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-10-03T07:10:22.074 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-10-03T07:10:22.074 INFO:teuthology.orchestra.run.smithi031.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0121288 s, 865 MB/s 2023-10-03T07:10:22.074 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-10-03T07:10:22.417 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi031:vg_nvme/lv_4 2023-10-03T07:10:22.668 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:10:23.044 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:22 smithi031 ceph-mon[107170]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:23.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:22 smithi139 ceph-mon[109290]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:22 smithi114 ceph-mon[109024]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:24.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:23 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:24.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:23 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:24.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:23 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:24.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:23 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:24.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:23 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:24.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:23 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:23 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:23 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:23 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:24.833 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:24 smithi031 ceph-mon[107170]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:24.833 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:24 smithi031 ceph-mon[107170]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:25.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:24 smithi139 ceph-mon[109290]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:25.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:24 smithi139 ceph-mon[109290]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:25.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:24 smithi114 ceph-mon[109024]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:25.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:24 smithi114 ceph-mon[109024]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:26.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:25 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]: dispatch 2023-10-03T07:10:26.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:25 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]': finished 2023-10-03T07:10:26.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:25 smithi031 ceph-mon[107170]: osdmap e5: 1 total, 0 up, 1 in 2023-10-03T07:10:26.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:26.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:25 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]: dispatch 2023-10-03T07:10:26.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:25 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]': finished 2023-10-03T07:10:26.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:25 smithi139 ceph-mon[109290]: osdmap e5: 1 total, 0 up, 1 in 2023-10-03T07:10:26.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:26.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:25 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]: dispatch 2023-10-03T07:10:26.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:25 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/4294403959' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32d40a81-22e7-4a48-94e9-7a836422ea3b"}]': finished 2023-10-03T07:10:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:25 smithi114 ceph-mon[109024]: osdmap e5: 1 total, 0 up, 1 in 2023-10-03T07:10:26.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:27.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:26 smithi031 ceph-mon[107170]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:27.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:26 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3205819549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:27.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:26 smithi139 ceph-mon[109290]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:27.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:26 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3205819549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:26 smithi114 ceph-mon[109024]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:26 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3205819549' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:28.906 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:28 smithi031 ceph-mon[107170]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:29.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:28 smithi139 ceph-mon[109290]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:29.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:28 smithi114 ceph-mon[109024]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:30.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:30 smithi139 ceph-mon[109290]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:30.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:30 smithi031 ceph-mon[107170]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:30.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:30 smithi114 ceph-mon[109024]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:31.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-03T07:10:31.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:31.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:31 smithi031 ceph-mon[107170]: Deploying daemon osd.0 on smithi031 2023-10-03T07:10:31.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:31 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-03T07:10:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:31 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:31 smithi114 ceph-mon[109024]: Deploying daemon osd.0 on smithi031 2023-10-03T07:10:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:31 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2023-10-03T07:10:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:31 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:31 smithi139 ceph-mon[109290]: Deploying daemon osd.0 on smithi031 2023-10-03T07:10:32.539 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:32 smithi031 ceph-mon[107170]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:32.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:32 smithi114 ceph-mon[109024]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:32.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:32 smithi139 ceph-mon[109290]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:34.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:34 smithi031 ceph-mon[107170]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:34.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:34 smithi139 ceph-mon[109290]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:34 smithi114 ceph-mon[109024]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:35 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:35 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.681 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:35 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:35.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:35 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:35 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:35 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:35 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:35 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:35 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:36.673 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 0 on host 'smithi031' 2023-10-03T07:10:36.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:36.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:36.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:36.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:36 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:36.966 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:36 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:36.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:36 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.116 DEBUG:teuthology.orchestra.run.smithi031:osd.0> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.0.service 2023-10-03T07:10:37.119 INFO:tasks.cephadm:Deploying osd.1 on smithi031 with /dev/vg_nvme/lv_3... 2023-10-03T07:10:37.119 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-03T07:10:37.155 INFO:journalctl@ceph.osd.0.smithi031.stdout:-- Logs begin at Tue 2023-10-03 06:59:00 UTC. -- 2023-10-03T07:10:37.657 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:10:37.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:37 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:37 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.855 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:37 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:37.855 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:10:37 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:10:37.740+0000 7f3260cfc640 -1 osd.0 0 log_to_monitors true 2023-10-03T07:10:37.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:37 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:37 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:37 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:37 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:37 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:37 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:38.763 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:38 smithi031 ceph-mon[107170]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:38.764 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:38 smithi031 ceph-mon[107170]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-03T07:10:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:38 smithi114 ceph-mon[109024]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:38 smithi114 ceph-mon[109024]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-03T07:10:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-03T07:10:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-03T07:10:39.116 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10+0 records in 2023-10-03T07:10:39.116 INFO:teuthology.orchestra.run.smithi031.stderr:10+0 records out 2023-10-03T07:10:39.117 INFO:teuthology.orchestra.run.smithi031.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0173188 s, 605 MB/s 2023-10-03T07:10:39.117 INFO:teuthology.orchestra.run.smithi031.stderr:--> Zapping successful for: 2023-10-03T07:10:39.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:38 smithi139 ceph-mon[109290]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:39.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:38 smithi139 ceph-mon[109290]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2023-10-03T07:10:39.514 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi031:vg_nvme/lv_3 2023-10-03T07:10:39.769 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:10:39.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:10:39.630+0000 7f3256374700 -1 osd.0 0 waiting for initial osdmap 2023-10-03T07:10:39.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:10:39.633+0000 7f3251167700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:10:39.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-03T07:10:39.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: osdmap e6: 1 total, 0 up, 1 in 2023-10-03T07:10:39.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:39.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi031 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi031 to 17118M 2023-10-03T07:10:39.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:39.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:39.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:39 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: osdmap e6: 1 total, 0 up, 1 in 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi031 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:39.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi031 to 17118M 2023-10-03T07:10:39.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:39.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:39.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:39.995 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:39 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:40.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2023-10-03T07:10:40.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: osdmap e6: 1 total, 0 up, 1 in 2023-10-03T07:10:40.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi031 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi031 to 17118M 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:40.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:40.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:39 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:40.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:40 smithi031 ceph-mon[107170]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:40.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:40 smithi031 ceph-mon[107170]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:40.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:40 smithi031 ceph-mon[107170]: osdmap e7: 1 total, 0 up, 1 in 2023-10-03T07:10:40.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:40 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:40.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:40 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:40.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:40 smithi114 ceph-mon[109024]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:40.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:40 smithi114 ceph-mon[109024]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:40.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:40 smithi114 ceph-mon[109024]: osdmap e7: 1 total, 0 up, 1 in 2023-10-03T07:10:40.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:40 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:40.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:40 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:40 smithi139 ceph-mon[109290]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:41.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:40 smithi139 ceph-mon[109290]: from='osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:41.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:40 smithi139 ceph-mon[109290]: osdmap e7: 1 total, 0 up, 1 in 2023-10-03T07:10:41.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:40 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:40 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:10:41.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:10:41.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.959 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756] boot 2023-10-03T07:10:41.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: osdmap e8: 1 total, 1 up, 1 in 2023-10-03T07:10:41.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:41.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:41.960 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:10:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:10:41.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756] boot 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: osdmap e8: 1 total, 1 up, 1 in 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:41.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:42.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:10:42.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: osd.0 [v2:172.21.15.31:6802/1219341756,v1:172.21.15.31:6803/1219341756] boot 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: osdmap e8: 1 total, 1 up, 1 in 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:42.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:42 smithi114 ceph-mon[109024]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:42 smithi114 ceph-mon[109024]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:42 smithi114 ceph-mon[109024]: osdmap e9: 1 total, 1 up, 1 in 2023-10-03T07:10:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:42 smithi031 ceph-mon[107170]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:42 smithi031 ceph-mon[107170]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:42 smithi031 ceph-mon[107170]: osdmap e9: 1 total, 1 up, 1 in 2023-10-03T07:10:43.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:42 smithi139 ceph-mon[109290]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi031:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:43.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:42 smithi139 ceph-mon[109290]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-10-03T07:10:43.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:42 smithi139 ceph-mon[109290]: osdmap e9: 1 total, 1 up, 1 in 2023-10-03T07:10:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:43 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]: dispatch 2023-10-03T07:10:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:43 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]': finished 2023-10-03T07:10:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:43 smithi114 ceph-mon[109024]: osdmap e10: 2 total, 1 up, 2 in 2023-10-03T07:10:43.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:43 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:43.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:43 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1302911936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:44.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:43 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]: dispatch 2023-10-03T07:10:44.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:43 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]': finished 2023-10-03T07:10:44.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:43 smithi031 ceph-mon[107170]: osdmap e10: 2 total, 1 up, 2 in 2023-10-03T07:10:44.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:43 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:44.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:43 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1302911936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:44.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:43 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]: dispatch 2023-10-03T07:10:44.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:43 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1238441770' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32be39eb-7ab6-42a7-87f4-9e3a5b9684cf"}]': finished 2023-10-03T07:10:44.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:43 smithi139 ceph-mon[109290]: osdmap e10: 2 total, 1 up, 2 in 2023-10-03T07:10:44.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:43 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:44.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:43 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1302911936' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:10:45.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:44 smithi031 ceph-mon[107170]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:45.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:44 smithi139 ceph-mon[109290]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:45.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:44 smithi114 ceph-mon[109024]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:46.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:46 smithi031 ceph-mon[107170]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:47.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:46 smithi139 ceph-mon[109290]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:46 smithi114 ceph-mon[109024]: pgmap v38: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:49.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:48 smithi031 ceph-mon[107170]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-03T07:10:49.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:48 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:49.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:48 smithi139 ceph-mon[109290]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:49.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:48 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-03T07:10:49.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:48 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:48 smithi114 ceph-mon[109024]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:48 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2023-10-03T07:10:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:48 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:49.954 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:49 smithi031 ceph-mon[107170]: Deploying daemon osd.1 on smithi031 2023-10-03T07:10:50.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:49 smithi139 ceph-mon[109290]: Deploying daemon osd.1 on smithi031 2023-10-03T07:10:50.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:49 smithi114 ceph-mon[109024]: Deploying daemon osd.1 on smithi031 2023-10-03T07:10:51.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:50 smithi031 ceph-mon[107170]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:51.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:50 smithi139 ceph-mon[109290]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:51.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:50 smithi114 ceph-mon[109024]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:52.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:52 smithi031 ceph-mon[107170]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:52.724 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:52 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:52.724 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:52 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:52.724 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:52 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:52 smithi114 ceph-mon[109024]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:52 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:52 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:52 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:53.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:52 smithi139 ceph-mon[109290]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:53.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:52 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:53.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:52 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:53.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:52 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:53.787 INFO:teuthology.orchestra.run.smithi031.stdout:Created osd(s) 1 on host 'smithi031' 2023-10-03T07:10:54.189 DEBUG:teuthology.orchestra.run.smithi031:osd.1> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.1.service 2023-10-03T07:10:54.191 INFO:tasks.cephadm:Deploying osd.2 on smithi114 with /dev/vg_nvme/lv_4... 2023-10-03T07:10:54.192 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-03T07:10:54.228 INFO:journalctl@ceph.osd.1.smithi031.stdout:-- Logs begin at Tue 2023-10-03 06:59:00 UTC. -- 2023-10-03T07:10:54.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:54.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:54.313 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:54.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:54.314 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.416 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:10:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:54.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:54.494 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:54 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:54.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:54.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:54.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:54.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:10:54.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:54 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:55.064 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:10:54 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:10:54.839+0000 7f10d4906640 -1 osd.1 0 log_to_monitors true 2023-10-03T07:10:55.616 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-03T07:10:55.616 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-03T07:10:55.616 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2023-10-03T07:10:55.616 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2023-10-03T07:10:55.617 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112249 s, 934 MB/s 2023-10-03T07:10:55.617 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2023-10-03T07:10:55.648 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:55 smithi114 ceph-mon[109024]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.648 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:55 smithi114 ceph-mon[109024]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:55 smithi031 ceph-mon[107170]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.698 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:55 smithi031 ceph-mon[107170]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:55 smithi139 ceph-mon[109290]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:55 smithi139 ceph-mon[109290]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2023-10-03T07:10:55.960 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi114:vg_nvme/lv_4 2023-10-03T07:10:56.178 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:10:56.547 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-03T07:10:56.547 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: osdmap e11: 2 total, 1 up, 2 in 2023-10-03T07:10:56.547 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:56.547 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.547 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:56.548 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:10:56.386+0000 7f10c9f7e700 -1 osd.1 0 waiting for initial osdmap 2023-10-03T07:10:56.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:10:56.390+0000 7f10c4570700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: osdmap e11: 2 total, 1 up, 2 in 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:56.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: osdmap e11: 2 total, 1 up, 2 in 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]: dispatch 2023-10-03T07:10:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:10:56.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi031 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi031 to 8559M 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: osdmap e12: 2 total, 1 up, 2 in 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.712 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.713 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:57.713 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:57.713 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:57 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi031 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi031 to 8559M 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: osdmap e12: 2 total, 1 up, 2 in 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:57.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:57.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:57 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:57.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi031 2023-10-03T07:10:57.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi031 to 8559M 2023-10-03T07:10:57.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi031", "root=default"]}]': finished 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: osdmap e12: 2 total, 1 up, 2 in 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:10:57.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:57 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178] boot 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: osdmap e13: 2 total, 2 up, 2 in 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:58.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:58 smithi114 ceph-mon[109024]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:58.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178] boot 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: osdmap e13: 2 total, 2 up, 2 in 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:58.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:58 smithi031 ceph-mon[107170]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:10:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:10:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: from='client.24158 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:10:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: osd.1 [v2:172.21.15.31:6810/2944524178,v1:172.21.15.31:6811/2944524178] boot 2023-10-03T07:10:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: osdmap e13: 2 total, 2 up, 2 in 2023-10-03T07:10:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2023-10-03T07:10:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:58 smithi139 ceph-mon[109290]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2023-10-03T07:10:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: osdmap e14: 2 total, 2 up, 2 in 2023-10-03T07:10:59.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/3574150019' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]': finished 2023-10-03T07:10:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: osdmap e15: 3 total, 2 up, 3 in 2023-10-03T07:10:59.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:10:59 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: osdmap e14: 2 total, 2 up, 2 in 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/3574150019' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]': finished 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: osdmap e15: 3 total, 2 up, 3 in 2023-10-03T07:10:59.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:10:59 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:10:59.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: osdmap e14: 2 total, 2 up, 2 in 2023-10-03T07:10:59.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/3574150019' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]: dispatch 2023-10-03T07:10:59.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "485d9558-1877-4502-9ae4-d2da6612140a"}]': finished 2023-10-03T07:10:59.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: osdmap e15: 3 total, 2 up, 3 in 2023-10-03T07:10:59.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:10:59 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:00 smithi114 ceph-mon[109024]: pgmap v50: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:00 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/468000586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:00.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:00 smithi031 ceph-mon[107170]: pgmap v50: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:00.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:00 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/468000586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:00 smithi139 ceph-mon[109290]: pgmap v50: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:00 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/468000586' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:02.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:02 smithi031 ceph-mon[107170]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:02.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:02 smithi139 ceph-mon[109290]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:02.941 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:02 smithi114 ceph-mon[109024]: pgmap v51: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:04.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:04 smithi114 ceph-mon[109024]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:04.657 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:04 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-03T07:11:04.658 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:04 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:04.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:04 smithi031 ceph-mon[107170]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:04.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-03T07:11:04.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:04.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:04 smithi139 ceph-mon[109290]: pgmap v52: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:04.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2023-10-03T07:11:04.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:05.776 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:05 smithi114 ceph-mon[109024]: Deploying daemon osd.2 on smithi114 2023-10-03T07:11:05.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:05 smithi031 ceph-mon[107170]: Deploying daemon osd.2 on smithi114 2023-10-03T07:11:05.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:05 smithi139 ceph-mon[109290]: Deploying daemon osd.2 on smithi114 2023-10-03T07:11:06.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:06 smithi031 ceph-mon[107170]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:06.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:06 smithi139 ceph-mon[109290]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:06.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:06 smithi114 ceph-mon[109024]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:08.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:08 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:08 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:08 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:08.488 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:08 smithi114 ceph-mon[109024]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:08.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:08 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:08 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:08 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:08.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:08 smithi031 ceph-mon[107170]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:08.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:08 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:08 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:08.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:08 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:08.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:08 smithi139 ceph-mon[109290]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:09.255 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 2 on host 'smithi114' 2023-10-03T07:11:09.629 DEBUG:teuthology.orchestra.run.smithi114:osd.2> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.2.service 2023-10-03T07:11:09.632 INFO:tasks.cephadm:Deploying osd.3 on smithi114 with /dev/vg_nvme/lv_3... 2023-10-03T07:11:09.632 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-03T07:11:09.663 INFO:journalctl@ceph.osd.2.smithi114.stdout:-- Logs begin at Tue 2023-10-03 06:59:12 UTC. -- 2023-10-03T07:11:09.686 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.686 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.686 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:09.687 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:09.687 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.687 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.687 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.687 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:09 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:09 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:09.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:09.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:09 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:10.181 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:11:10.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:10 smithi114 ceph-mon[109024]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:10.744 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:10 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:11:10.548+0000 7f7915770640 -1 osd.2 0 log_to_monitors true 2023-10-03T07:11:10.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:10 smithi031 ceph-mon[107170]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:10.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:10 smithi139 ceph-mon[109290]: pgmap v55: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:11.459 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-03T07:11:11.459 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-03T07:11:11.460 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2023-10-03T07:11:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2023-10-03T07:11:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0124975 s, 839 MB/s 2023-10-03T07:11:11.460 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2023-10-03T07:11:11.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:11 smithi114 ceph-mon[109024]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:11.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:11 smithi114 ceph-mon[109024]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:11.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:11 smithi031 ceph-mon[107170]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:11.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:11 smithi031 ceph-mon[107170]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:11.857 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi114:vg_nvme/lv_3 2023-10-03T07:11:11.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:11 smithi139 ceph-mon[109290]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:11.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:11 smithi139 ceph-mon[109290]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2023-10-03T07:11:12.112 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:11:12.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:11:12.451+0000 7f790ade8700 -1 osd.2 0 waiting for initial osdmap 2023-10-03T07:11:12.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:11:12.454+0000 7f79053da700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:11:12.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-03T07:11:12.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: osdmap e16: 3 total, 2 up, 3 in 2023-10-03T07:11:12.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:12.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi114 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi114 to 21209M 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:12.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:12.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: osdmap e16: 3 total, 2 up, 3 in 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi114 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi114 to 21209M 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:12.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: osdmap e16: 3 total, 2 up, 3 in 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi114 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi114 to 21209M 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:12.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:13.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: osdmap e17: 3 total, 2 up, 3 in 2023-10-03T07:11:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:13.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:13.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:13 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:13.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: osdmap e17: 3 total, 2 up, 3 in 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:13.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:13 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:13.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:13.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: osdmap e17: 3 total, 2 up, 3 in 2023-10-03T07:11:13.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:13.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:13.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:13.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:13 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:11:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:11:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567] boot 2023-10-03T07:11:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: osdmap e18: 3 total, 3 up, 3 in 2023-10-03T07:11:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: pgmap v60: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:14.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:14 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:14.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567] boot 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: osdmap e18: 3 total, 3 up, 3 in 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: pgmap v60: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:14 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:11:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:11:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: from='client.24185 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: osd.2 [v2:172.21.15.114:6800/1195543567,v1:172.21.15.114:6801/1195543567] boot 2023-10-03T07:11:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: osdmap e18: 3 total, 3 up, 3 in 2023-10-03T07:11:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2023-10-03T07:11:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: pgmap v60: 0 pgs: ; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:14.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:14 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: osdmap e19: 3 total, 3 up, 3 in 2023-10-03T07:11:15.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/2695531591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]': finished 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: osdmap e20: 4 total, 3 up, 4 in 2023-10-03T07:11:15.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:15.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:15 smithi031 sudo[124606]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:15.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:15 smithi031 sudo[124606]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:15.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:15 smithi031 sudo[124606]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:15.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:15 smithi031 sudo[124606]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: osdmap e19: 3 total, 3 up, 3 in 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/2695531591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]': finished 2023-10-03T07:11:15.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: osdmap e20: 4 total, 3 up, 4 in 2023-10-03T07:11:15.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: osdmap e19: 3 total, 3 up, 3 in 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/2695531591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]: dispatch 2023-10-03T07:11:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2023-10-03T07:11:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "56329ffe-90ad-471a-9493-219054d65a29"}]': finished 2023-10-03T07:11:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: osdmap e20: 4 total, 3 up, 4 in 2023-10-03T07:11:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:16.456 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124609]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:16.456 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124609]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:16.456 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124609]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:16.456 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124609]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:16 smithi114 ceph-mon[109024]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:16 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/1304320005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:16 smithi114 ceph-mon[109024]: osdmap e21: 4 total, 3 up, 4 in 2023-10-03T07:11:16.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:16.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:16.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:16 smithi031 ceph-mon[107170]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:16 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/1304320005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:16 smithi031 ceph-mon[107170]: osdmap e21: 4 total, 3 up, 4 in 2023-10-03T07:11:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:16.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124612]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:16.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124612]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:16.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124612]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:16.817 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:16 smithi031 sudo[124612]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:16 smithi139 ceph-mon[109290]: pgmap v63: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:16 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/1304320005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:16 smithi139 ceph-mon[109290]: osdmap e21: 4 total, 3 up, 4 in 2023-10-03T07:11:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:16.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:17.316 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:17 smithi031 sudo[124615]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:17.317 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:17 smithi031 sudo[124615]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:17.317 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:17 smithi031 sudo[124615]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:17.317 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:11:17 smithi031 sudo[124615]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116123]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116123]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116123]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116123]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116128]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116128]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116128]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:17.531 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:11:17 smithi114 sudo[116128]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:18.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124618]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:18.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124618]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:18.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124618]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:18.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124618]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:18.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:18 smithi114 ceph-mon[109024]: mgrmap e14: x(active, since 107s) 2023-10-03T07:11:18.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:18.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:18 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124621]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124621]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124621]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 sudo[124621]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 ceph-mon[107170]: mgrmap e14: x(active, since 107s) 2023-10-03T07:11:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:18.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:18 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 ceph-mon[109290]: mgrmap e14: x(active, since 107s) 2023-10-03T07:11:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111470]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:18.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:18.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111470]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:18.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111470]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:19.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111473]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:19.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:19.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111473]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:19.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:18 smithi139 sudo[111473]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:19.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116449]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/smartctl -x --json=o /dev/nvme0n1 2023-10-03T07:11:19.510 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116449]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:19.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116449]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:19.511 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116449]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:19 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:19 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.884 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116471]: ceph : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2023-10-03T07:11:19.884 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116471]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2023-10-03T07:11:19.884 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116471]: pam_unix(sudo:session): session opened for user root by (uid=0) 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 sudo[116471]: pam_unix(sudo:session): session closed for user root 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: pgmap v65: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:19.885 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2023-10-03T07:11:19.886 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2023-10-03T07:11:19.886 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:19 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:20.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:20 smithi114 ceph-mon[109024]: Deploying daemon osd.3 on smithi114 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:20.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:20 smithi031 ceph-mon[107170]: Deploying daemon osd.3 on smithi114 2023-10-03T07:11:20.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2023-10-03T07:11:20.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: pgmap v66: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:20.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2023-10-03T07:11:20.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2023-10-03T07:11:20.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:20.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:20 smithi139 ceph-mon[109290]: Deploying daemon osd.3 on smithi114 2023-10-03T07:11:22.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:22 smithi114 ceph-mon[109024]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:22.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:22 smithi031 ceph-mon[107170]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:22.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:22 smithi139 ceph-mon[109290]: pgmap v67: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2023-10-03T07:11:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:24 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:24 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:24 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:24.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:24 smithi114 ceph-mon[109024]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-10-03T07:11:24.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:24 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:24 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:24 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:24.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:24 smithi031 ceph-mon[107170]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-10-03T07:11:24.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:24 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:24 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:24.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:24 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:24.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:24 smithi139 ceph-mon[109290]: pgmap v68: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 4.5 KiB/s rd, 196 KiB/s wr, 13 op/s 2023-10-03T07:11:25.055 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 3 on host 'smithi114' 2023-10-03T07:11:25.435 DEBUG:teuthology.orchestra.run.smithi114:osd.3> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.3.service 2023-10-03T07:11:25.438 INFO:tasks.cephadm:Deploying osd.4 on smithi114 with /dev/vg_nvme/lv_2... 2023-10-03T07:11:25.439 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-03T07:11:25.469 INFO:journalctl@ceph.osd.3.smithi114.stdout:-- Logs begin at Tue 2023-10-03 06:59:12 UTC. -- 2023-10-03T07:11:25.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.492 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:25.493 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:25 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:25.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:25 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:25.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:25 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:25.979 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:11:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:26 smithi114 ceph-mon[109024]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-03T07:11:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:26 smithi114 ceph-mon[109024]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:26.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:26 smithi114 ceph-mon[109024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:26.744 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:11:26 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:11:26.368+0000 7efd5d530640 -1 osd.3 0 log_to_monitors true 2023-10-03T07:11:26.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:26 smithi031 ceph-mon[107170]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-03T07:11:26.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:26 smithi031 ceph-mon[107170]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:26.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:26 smithi031 ceph-mon[107170]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:26 smithi139 ceph-mon[109290]: pgmap v69: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 131 KiB/s wr, 9 op/s 2023-10-03T07:11:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:26 smithi139 ceph-mon[109290]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:26 smithi139 ceph-mon[109290]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2023-10-03T07:11:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-03T07:11:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-03T07:11:27.528 INFO:teuthology.orchestra.run.smithi114.stderr: stderr: 10+0 records in 2023-10-03T07:11:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:10+0 records out 2023-10-03T07:11:27.528 INFO:teuthology.orchestra.run.smithi114.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133776 s, 784 MB/s 2023-10-03T07:11:27.529 INFO:teuthology.orchestra.run.smithi114.stderr:--> Zapping successful for: 2023-10-03T07:11:27.743 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:11:27.470+0000 7efd52ba8700 -1 osd.3 0 waiting for initial osdmap 2023-10-03T07:11:27.743 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:11:27.473+0000 7efd4e99d700 -1 osd.3 23 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:11:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-mon[109024]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-03T07:11:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-mon[109024]: osdmap e22: 4 total, 3 up, 4 in 2023-10-03T07:11:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-mon[109024]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:27 smithi114 ceph-mon[109024]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:27 smithi031 ceph-mon[107170]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-03T07:11:27.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:27 smithi031 ceph-mon[107170]: osdmap e22: 4 total, 3 up, 4 in 2023-10-03T07:11:27.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:27 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:27.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:27 smithi031 ceph-mon[107170]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:27 smithi031 ceph-mon[107170]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:27 smithi139 ceph-mon[109290]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2023-10-03T07:11:27.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:27 smithi139 ceph-mon[109290]: osdmap e22: 4 total, 3 up, 4 in 2023-10-03T07:11:27.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:27 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:27.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:27 smithi139 ceph-mon[109290]: from='osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:27 smithi139 ceph-mon[109290]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:27.913 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi114:vg_nvme/lv_2 2023-10-03T07:11:28.168 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:11:28.604 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:28.604 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: osdmap e23: 4 total, 3 up, 4 in 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi114 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi114 to 10604M 2023-10-03T07:11:28.605 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599] boot 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: osdmap e24: 4 total, 4 up, 4 in 2023-10-03T07:11:28.606 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: osdmap e23: 4 total, 3 up, 4 in 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi114 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi114 to 10604M 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599] boot 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: osdmap e24: 4 total, 4 up, 4 in 2023-10-03T07:11:28.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: osdmap e23: 4 total, 3 up, 4 in 2023-10-03T07:11:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 83 MiB used, 268 GiB / 268 GiB avail; 2.2 KiB/s rd, 98 KiB/s wr, 6 op/s 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi114 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi114 to 10604M 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: osd.3 [v2:172.21.15.114:6808/3292587599,v1:172.21.15.114:6809/3292587599] boot 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: osdmap e24: 4 total, 4 up, 4 in 2023-10-03T07:11:28.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:29.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:29 smithi114 ceph-mon[109024]: osdmap e25: 4 total, 4 up, 4 in 2023-10-03T07:11:29.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:11:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:11:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:29 smithi031 ceph-mon[107170]: osdmap e25: 4 total, 4 up, 4 in 2023-10-03T07:11:29.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:11:29.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:11:29.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:29.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:29.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:29.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:29 smithi139 ceph-mon[109290]: osdmap e25: 4 total, 4 up, 4 in 2023-10-03T07:11:30.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:30 smithi114 ceph-mon[109024]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:30.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:30 smithi114 ceph-mon[109024]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:30.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:30 smithi114 ceph-mon[109024]: osdmap e26: 4 total, 4 up, 4 in 2023-10-03T07:11:30.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:30 smithi031 ceph-mon[107170]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:30.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:30 smithi031 ceph-mon[107170]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:30.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:30 smithi031 ceph-mon[107170]: osdmap e26: 4 total, 4 up, 4 in 2023-10-03T07:11:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:30 smithi139 ceph-mon[109290]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi114:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:30 smithi139 ceph-mon[109290]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:30 smithi139 ceph-mon[109290]: osdmap e26: 4 total, 4 up, 4 in 2023-10-03T07:11:31.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/2371515492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]': finished 2023-10-03T07:11:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: osdmap e27: 5 total, 4 up, 5 in 2023-10-03T07:11:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:31.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:31 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/1244819420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:31.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/2371515492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]': finished 2023-10-03T07:11:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: osdmap e27: 5 total, 4 up, 5 in 2023-10-03T07:11:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:31 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/1244819420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/2371515492' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]: dispatch 2023-10-03T07:11:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c2014159-a7d2-4fd5-b06e-e08543bd98b8"}]': finished 2023-10-03T07:11:31.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: osdmap e27: 5 total, 4 up, 5 in 2023-10-03T07:11:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:31 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/1244819420' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:33.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:32 smithi139 ceph-mon[109290]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:33.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:32 smithi139 ceph-mon[109290]: osdmap e28: 5 total, 4 up, 5 in 2023-10-03T07:11:33.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:32 smithi114 ceph-mon[109024]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:32 smithi114 ceph-mon[109024]: osdmap e28: 5 total, 4 up, 5 in 2023-10-03T07:11:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:33.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:32 smithi031 ceph-mon[107170]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:33.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:32 smithi031 ceph-mon[107170]: osdmap e28: 5 total, 4 up, 5 in 2023-10-03T07:11:33.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:33 smithi114 ceph-mon[109024]: osdmap e29: 5 total, 4 up, 5 in 2023-10-03T07:11:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:33 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:34.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:33 smithi139 ceph-mon[109290]: osdmap e29: 5 total, 4 up, 5 in 2023-10-03T07:11:34.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:33 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:34.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:33 smithi031 ceph-mon[107170]: osdmap e29: 5 total, 4 up, 5 in 2023-10-03T07:11:34.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:33 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:35.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:34 smithi139 ceph-mon[109290]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:11:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:34 smithi114 ceph-mon[109024]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:11:35.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:34 smithi031 ceph-mon[107170]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:11:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:35 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-03T07:11:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:35 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:36.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:35 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-03T07:11:36.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:35 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:36.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:35 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2023-10-03T07:11:36.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:35 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:37.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:36 smithi139 ceph-mon[109290]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2023-10-03T07:11:37.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:36 smithi139 ceph-mon[109290]: Deploying daemon osd.4 on smithi114 2023-10-03T07:11:37.138 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:36 smithi114 ceph-mon[109024]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2023-10-03T07:11:37.138 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:36 smithi114 ceph-mon[109024]: Deploying daemon osd.4 on smithi114 2023-10-03T07:11:37.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:36 smithi031 ceph-mon[107170]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 113 KiB/s, 0 objects/s recovering 2023-10-03T07:11:37.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:36 smithi031 ceph-mon[107170]: Deploying daemon osd.4 on smithi114 2023-10-03T07:11:39.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:38 smithi139 ceph-mon[109290]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-03T07:11:39.154 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:38 smithi114 ceph-mon[109024]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-03T07:11:39.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:38 smithi031 ceph-mon[107170]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 109 MiB used, 358 GiB / 358 GiB avail; 87 KiB/s, 0 objects/s recovering 2023-10-03T07:11:40.236 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:40 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:40 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:40 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:40.237 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:40 smithi114 ceph-mon[109024]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-03T07:11:40.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:40 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:40 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:40 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:40.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:40 smithi031 ceph-mon[107170]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-03T07:11:40.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:40 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:40 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:40.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:40 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:40.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:40 smithi139 ceph-mon[109290]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2023-10-03T07:11:41.374 INFO:teuthology.orchestra.run.smithi114.stdout:Created osd(s) 4 on host 'smithi114' 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.667 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.668 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:41 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:41.733 DEBUG:teuthology.orchestra.run.smithi114:osd.4> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.4.service 2023-10-03T07:11:41.736 INFO:tasks.cephadm:Deploying osd.5 on smithi139 with /dev/vg_nvme/lv_4... 2023-10-03T07:11:41.736 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2023-10-03T07:11:41.766 INFO:journalctl@ceph.osd.4.smithi114.stdout:-- Logs begin at Tue 2023-10-03 06:59:12 UTC. -- 2023-10-03T07:11:41.767 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.767 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.767 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:41.767 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:41.767 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.768 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.768 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.768 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:41 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:41.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:41.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:41 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:41.960 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:11:42.489 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:11:42 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:11:42.459+0000 7fc0b9a7b640 -1 osd.4 0 log_to_monitors true 2023-10-03T07:11:42.659 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:42 smithi139 ceph-mon[109290]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:42.659 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:42 smithi139 ceph-mon[109290]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:42.659 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:42 smithi139 ceph-mon[109290]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:42 smithi114 ceph-mon[109024]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:42 smithi114 ceph-mon[109024]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:42.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:42 smithi114 ceph-mon[109024]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:42.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:42 smithi031 ceph-mon[107170]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:42.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:42 smithi031 ceph-mon[107170]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:42.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:42 smithi031 ceph-mon[107170]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_4 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112263 s, 934 MB/s 2023-10-03T07:11:43.084 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-10-03T07:11:43.417 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi139:vg_nvme/lv_4 2023-10-03T07:11:43.677 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: osdmap e30: 5 total, 4 up, 5 in 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi114 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi114 to 7069M 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:44.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:44.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.624 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:44 smithi139 ceph-mon[109290]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-03T07:11:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: osdmap e30: 5 total, 4 up, 5 in 2023-10-03T07:11:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:44.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi114 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi114 to 7069M 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.746 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-mon[109024]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:44.747 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:11:44.383+0000 7fc0af0f3700 -1 osd.4 0 waiting for initial osdmap 2023-10-03T07:11:44.747 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:11:44 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:11:44.390+0000 7fc0a86e3700 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: osdmap e30: 5 total, 4 up, 5 in 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]: dispatch 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi114 2023-10-03T07:11:44.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi114 to 7069M 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:44.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:44.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:44.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:44 smithi031 ceph-mon[107170]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: osdmap e31: 5 total, 4 up, 5 in 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:45.686 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:45.687 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:45.687 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:45.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:45.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: osdmap e31: 5 total, 4 up, 5 in 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:45.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi114", "root=default"]}]': finished 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: osdmap e31: 5 total, 4 up, 5 in 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='client.24214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:11:45.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:11:45.818 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:46.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497] boot 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: osdmap e32: 5 total, 5 up, 5 in 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:46 smithi114 ceph-mon[109024]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497] boot 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: osdmap e32: 5 total, 5 up, 5 in 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:46 smithi031 ceph-mon[107170]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: osd.4 [v2:172.21.15.114:6816/1956269497,v1:172.21.15.114:6817/1956269497] boot 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: osdmap e32: 5 total, 5 up, 5 in 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2023-10-03T07:11:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:46 smithi139 ceph-mon[109290]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2023-10-03T07:11:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/2818525176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]': finished 2023-10-03T07:11:47.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: osdmap e33: 6 total, 5 up, 6 in 2023-10-03T07:11:47.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:11:47.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:47 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/3347308133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:47.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/2818525176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]': finished 2023-10-03T07:11:47.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: osdmap e33: 6 total, 5 up, 6 in 2023-10-03T07:11:47.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:11:47.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:47 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/3347308133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:47.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/2818525176' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]: dispatch 2023-10-03T07:11:47.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "16d7dda5-89e5-440e-ba54-4ec47bd79cb2"}]': finished 2023-10-03T07:11:47.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: osdmap e33: 6 total, 5 up, 6 in 2023-10-03T07:11:47.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:11:47.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:47 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/3347308133' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:11:48.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:48 smithi114 ceph-mon[109024]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:48.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:48 smithi031 ceph-mon[107170]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:48.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:48 smithi139 ceph-mon[109290]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:50.678 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:50 smithi139 ceph-mon[109290]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:50.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:50 smithi114 ceph-mon[109024]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:50.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:50 smithi031 ceph-mon[107170]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:51.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:51 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-03T07:11:51.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:51 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:51.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:51 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-03T07:11:51.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:51 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:51 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2023-10-03T07:11:51.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:51 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:52.793 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:52 smithi139 ceph-mon[109290]: Deploying daemon osd.5 on smithi139 2023-10-03T07:11:52.793 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:52 smithi139 ceph-mon[109290]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:52.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:52 smithi031 ceph-mon[107170]: Deploying daemon osd.5 on smithi139 2023-10-03T07:11:52.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:52 smithi031 ceph-mon[107170]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:52 smithi114 ceph-mon[109024]: Deploying daemon osd.5 on smithi139 2023-10-03T07:11:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:52 smithi114 ceph-mon[109024]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:54.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:54 smithi031 ceph-mon[107170]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:54.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:54 smithi139 ceph-mon[109290]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:54 smithi114 ceph-mon[109024]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:56.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:55 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:55 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:55 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:56.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:55 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:55 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:55 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:55 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:55 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:55 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:56.689 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 5 on host 'smithi139' 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:56.860 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.861 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.861 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:56.861 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:56 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:57.011 DEBUG:teuthology.orchestra.run.smithi139:osd.5> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.5.service 2023-10-03T07:11:57.014 INFO:tasks.cephadm:Deploying osd.6 on smithi139 with /dev/vg_nvme/lv_3... 2023-10-03T07:11:57.015 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2023-10-03T07:11:57.045 INFO:journalctl@ceph.osd.5.smithi139.stdout:-- Logs begin at Tue 2023-10-03 06:59:09 UTC. -- 2023-10-03T07:11:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:56 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:57.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:56 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:11:57.566 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:11:58.343 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:11:58.027+0000 7f40d68a5640 -1 osd.5 0 log_to_monitors true 2023-10-03T07:11:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:58 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_3 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112609 s, 931 MB/s 2023-10-03T07:11:58.932 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:59.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:59.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:59.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:58 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:11:59.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:58 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:11:59.335 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi139:vg_nvme/lv_3 2023-10-03T07:11:59.549 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:12:00.021 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:11:59.832+0000 7f40cbf1d700 -1 osd.5 0 waiting for initial osdmap 2023-10-03T07:12:00.021 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:11:59.837+0000 7f40c550d700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:12:00.021 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi139 2023-10-03T07:12:00.021 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi139 to 21209M 2023-10-03T07:12:00.021 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-03T07:12:00.022 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: osdmap e34: 6 total, 5 up, 6 in 2023-10-03T07:12:00.022 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:00.022 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:00.022 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:11:59 smithi139 ceph-mon[109290]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi139 2023-10-03T07:12:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi139 to 21209M 2023-10-03T07:12:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-03T07:12:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: osdmap e34: 6 total, 5 up, 6 in 2023-10-03T07:12:00.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:00.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:00.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:11:59 smithi114 ceph-mon[109024]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi139 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi139 to 21209M 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: osdmap e34: 6 total, 5 up, 6 in 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: from='osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:00.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:11:59 smithi031 ceph-mon[107170]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:01.105 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:01.105 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:01.105 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: osdmap e35: 6 total, 5 up, 6 in 2023-10-03T07:12:01.105 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.105 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.106 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:01.106 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:01.106 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:00 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:01.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: osdmap e35: 6 total, 5 up, 6 in 2023-10-03T07:12:01.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:01.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:01.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:00 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:01.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: osdmap e35: 6 total, 5 up, 6 in 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:01.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:00 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010] boot 2023-10-03T07:12:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: osdmap e36: 6 total, 6 up, 6 in 2023-10-03T07:12:01.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:01 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:12:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:12:02.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010] boot 2023-10-03T07:12:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: osdmap e36: 6 total, 6 up, 6 in 2023-10-03T07:12:02.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:01 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:02.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: osd.5 [v2:172.21.15.139:6800/2746087010,v1:172.21.15.139:6801/2746087010] boot 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: osdmap e36: 6 total, 6 up, 6 in 2023-10-03T07:12:02.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:01 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: osdmap e37: 6 total, 6 up, 6 in 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/2979143477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]': finished 2023-10-03T07:12:03.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: osdmap e38: 7 total, 6 up, 7 in 2023-10-03T07:12:03.123 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:02 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: osdmap e37: 6 total, 6 up, 6 in 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/2979143477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]': finished 2023-10-03T07:12:03.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: osdmap e38: 7 total, 6 up, 7 in 2023-10-03T07:12:03.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:02 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: osdmap e37: 6 total, 6 up, 6 in 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/2979143477' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]: dispatch 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9938d1fa-de80-46cc-a45a-fd4761b4ca49"}]': finished 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: osdmap e38: 7 total, 6 up, 7 in 2023-10-03T07:12:03.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:02 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:04.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:03 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/149795306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:04.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:03 smithi139 ceph-mon[109290]: osdmap e39: 7 total, 6 up, 7 in 2023-10-03T07:12:04.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:03 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:03 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/149795306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:04.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:03 smithi114 ceph-mon[109024]: osdmap e39: 7 total, 6 up, 7 in 2023-10-03T07:12:04.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:03 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:04.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:03 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/149795306' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:04.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:03 smithi031 ceph-mon[107170]: osdmap e39: 7 total, 6 up, 7 in 2023-10-03T07:12:04.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:03 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:05.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:04 smithi139 ceph-mon[109290]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 158 KiB/s, 0 objects/s recovering 2023-10-03T07:12:05.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:04 smithi139 ceph-mon[109290]: osdmap e40: 7 total, 6 up, 7 in 2023-10-03T07:12:05.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:04 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:05.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:04 smithi114 ceph-mon[109024]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 158 KiB/s, 0 objects/s recovering 2023-10-03T07:12:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:04 smithi114 ceph-mon[109024]: osdmap e40: 7 total, 6 up, 7 in 2023-10-03T07:12:05.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:04 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:05.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:04 smithi031 ceph-mon[107170]: pgmap v106: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 158 KiB/s, 0 objects/s recovering 2023-10-03T07:12:05.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:04 smithi031 ceph-mon[107170]: osdmap e40: 7 total, 6 up, 7 in 2023-10-03T07:12:05.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:04 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:07.120 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:06 smithi139 ceph-mon[109290]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:12:07.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:06 smithi114 ceph-mon[109024]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:12:07.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:06 smithi031 ceph-mon[107170]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2023-10-03T07:12:08.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:07 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-03T07:12:08.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:07 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:08.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:07 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-03T07:12:08.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:07 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:08.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:07 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2023-10-03T07:12:08.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:07 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:09.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:08 smithi139 ceph-mon[109290]: Deploying daemon osd.6 on smithi139 2023-10-03T07:12:09.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:08 smithi139 ceph-mon[109290]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-03T07:12:09.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:08 smithi114 ceph-mon[109024]: Deploying daemon osd.6 on smithi139 2023-10-03T07:12:09.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:08 smithi114 ceph-mon[109024]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-03T07:12:09.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:08 smithi031 ceph-mon[107170]: Deploying daemon osd.6 on smithi139 2023-10-03T07:12:09.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:08 smithi031 ceph-mon[107170]: pgmap v109: 1 pgs: 1 peering; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 102 KiB/s, 0 objects/s recovering 2023-10-03T07:12:10.656 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:10 smithi139 ceph-mon[109290]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-10-03T07:12:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:10 smithi114 ceph-mon[109024]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-10-03T07:12:10.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:10 smithi031 ceph-mon[107170]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2023-10-03T07:12:12.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:11 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:12.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:11 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:11 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:12.676 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 6 on host 'smithi139' 2023-10-03T07:12:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-10-03T07:12:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:12.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:12 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:13.057 DEBUG:teuthology.orchestra.run.smithi139:osd.6> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.6.service 2023-10-03T07:12:13.059 INFO:tasks.cephadm:Deploying osd.7 on smithi139 with /dev/vg_nvme/lv_2... 2023-10-03T07:12:13.060 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:12 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:13.089 INFO:journalctl@ceph.osd.6.smithi139.stdout:-- Logs begin at Tue 2023-10-03 06:59:09 UTC. -- 2023-10-03T07:12:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail; 71 KiB/s, 0 objects/s recovering 2023-10-03T07:12:13.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:13.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:12 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:13.567 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:12:14.122 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:12:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:12:14.025+0000 7fce15041640 -1 osd.6 0 log_to_monitors true 2023-10-03T07:12:14.970 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping: /dev/vg_nvme/lv_2 2023-10-03T07:12:14.970 INFO:teuthology.orchestra.run.smithi139.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2023-10-03T07:12:14.970 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10+0 records in 2023-10-03T07:12:14.970 INFO:teuthology.orchestra.run.smithi139.stderr:10+0 records out 2023-10-03T07:12:14.971 INFO:teuthology.orchestra.run.smithi139.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126565 s, 828 MB/s 2023-10-03T07:12:14.971 INFO:teuthology.orchestra.run.smithi139.stderr:--> Zapping successful for: 2023-10-03T07:12:15.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:14 smithi031 ceph-mon[107170]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:15.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:14 smithi031 ceph-mon[107170]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:14 smithi031 ceph-mon[107170]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:14 smithi139 ceph-mon[109290]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:15.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:14 smithi139 ceph-mon[109290]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:14 smithi139 ceph-mon[109290]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:14 smithi114 ceph-mon[109024]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:14 smithi114 ceph-mon[109024]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:14 smithi114 ceph-mon[109024]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2023-10-03T07:12:15.293 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph orch daemon add osd smithi139:vg_nvme/lv_2 2023-10-03T07:12:15.509 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:12:15.871 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:12:15.788+0000 7fce0a6b9700 -1 osd.6 0 waiting for initial osdmap 2023-10-03T07:12:15.871 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:12:15.793+0000 7fce05cad700 -1 osd.6 42 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:12:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-03T07:12:15.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: osdmap e41: 7 total, 6 up, 7 in 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi139 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi139 to 10604M 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:15.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:15.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:15 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: osdmap e41: 7 total, 6 up, 7 in 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi139 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi139 to 10604M 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:16.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:15 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2023-10-03T07:12:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: osdmap e41: 7 total, 6 up, 7 in 2023-10-03T07:12:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:16.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi139 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi139 to 10604M 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:16.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:15 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: osdmap e42: 7 total, 6 up, 7 in 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:17.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:16 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:17.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:17.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: osdmap e42: 7 total, 6 up, 7 in 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:17.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:16 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: osdmap e42: 7 total, 6 up, 7 in 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:17.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2023-10-03T07:12:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2023-10-03T07:12:17.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:16 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:18.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:12:18.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:12:18.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:18.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427] boot 2023-10-03T07:12:18.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: osdmap e43: 7 total, 7 up, 7 in 2023-10-03T07:12:18.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:17 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:18.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:12:18.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:12:18.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:18.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427] boot 2023-10-03T07:12:18.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: osdmap e43: 7 total, 7 up, 7 in 2023-10-03T07:12:18.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:17 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:18.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:12:18.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:12:18.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: from='client.24268 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi139:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:12:18.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: osd.6 [v2:172.21.15.139:6808/1462301427,v1:172.21.15.139:6809/1462301427] boot 2023-10-03T07:12:18.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: osdmap e43: 7 total, 7 up, 7 in 2023-10-03T07:12:18.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:17 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: osdmap e44: 7 total, 7 up, 7 in 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/4079320852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]': finished 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: osdmap e45: 8 total, 7 up, 8 in 2023-10-03T07:12:19.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:18 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:19.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: osdmap e44: 7 total, 7 up, 7 in 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/4079320852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]': finished 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: osdmap e45: 8 total, 7 up, 8 in 2023-10-03T07:12:19.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:18 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2023-10-03T07:12:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: osdmap e44: 7 total, 7 up, 7 in 2023-10-03T07:12:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/4079320852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]: dispatch 2023-10-03T07:12:19.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "20b1bc93-429a-4914-842f-243ac5be829a"}]': finished 2023-10-03T07:12:19.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: osdmap e45: 8 total, 7 up, 8 in 2023-10-03T07:12:19.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:18 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:20.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:19 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/2926304820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:20.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:19 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/2926304820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:20.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:19 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/2926304820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2023-10-03T07:12:21.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:20 smithi031 ceph-mon[107170]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:21.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:20 smithi139 ceph-mon[109290]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:21.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:20 smithi114 ceph-mon[109024]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:23.051 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:22 smithi139 ceph-mon[109290]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:23.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:22 smithi031 ceph-mon[107170]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:23.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:22 smithi114 ceph-mon[109024]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:24.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:23 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-03T07:12:24.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:23 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:24.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:23 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-03T07:12:24.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:23 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:23 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2023-10-03T07:12:24.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:23 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:25.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:24 smithi031 ceph-mon[107170]: Deploying daemon osd.7 on smithi139 2023-10-03T07:12:25.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:24 smithi031 ceph-mon[107170]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:25.076 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:24 smithi139 ceph-mon[109290]: Deploying daemon osd.7 on smithi139 2023-10-03T07:12:25.077 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:24 smithi139 ceph-mon[109290]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:25.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:24 smithi114 ceph-mon[109024]: Deploying daemon osd.7 on smithi139 2023-10-03T07:12:25.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:24 smithi114 ceph-mon[109024]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:27.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:26 smithi031 ceph-mon[107170]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:27.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:26 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:27.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:26 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:27.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:26 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:27.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:26 smithi139 ceph-mon[109290]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:27.086 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:26 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:27.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:26 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:27.087 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:26 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:26 smithi114 ceph-mon[109024]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:27.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:26 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:26 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:27.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:26 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.588 INFO:teuthology.orchestra.run.smithi139.stdout:Created osd(s) 7 on host 'smithi139' 2023-10-03T07:12:28.616 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.616 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.616 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:28.616 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:28.617 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.617 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:28 smithi139 ceph-mon[109290]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:28 smithi114 ceph-mon[109024]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:28 smithi031 ceph-mon[107170]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:28.980 DEBUG:teuthology.orchestra.run.smithi139:osd.7> sudo journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.7.service 2023-10-03T07:12:28.982 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2023-10-03T07:12:28.983 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd stat -f json 2023-10-03T07:12:29.013 INFO:journalctl@ceph.osd.7.smithi139.stdout:-- Logs begin at Tue 2023-10-03 06:59:09 UTC. -- 2023-10-03T07:12:29.234 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:29.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:29.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:12:29 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:12:29.675+0000 7f715fa8f640 -1 osd.7 0 log_to_monitors true 2023-10-03T07:12:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:29.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:12:30.425 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1879879045' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:30.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:30.568 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:30 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.801 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":45,"num_osds":8,"num_up_osds":7,"osd_up_since":1696317136,"num_in_osds":8,"osd_in_since":1696317138,"num_remapped_pgs":0} 2023-10-03T07:12:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1879879045' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:30.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:30 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:30.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1879879045' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:12:30.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:30 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:12:31.802 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd stat -f json 2023-10-03T07:12:31.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: Detected new or changed devices on smithi139 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: Adjusting osd_memory_target on smithi139 to 7069M 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: osdmap e46: 8 total, 7 up, 8 in 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:31.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:31 smithi031 ceph-mon[107170]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:31.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:12:31.561+0000 7f7155107700 -1 osd.7 0 waiting for initial osdmap 2023-10-03T07:12:31.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:12:31.568+0000 7f714fefa700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: Detected new or changed devices on smithi139 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: Adjusting osd_memory_target on smithi139 to 7069M 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: osdmap e46: 8 total, 7 up, 8 in 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:31.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:31.873 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:31 smithi139 ceph-mon[109290]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:31.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: Detected new or changed devices on smithi139 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: Adjusting osd_memory_target on smithi139 to 7069M 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: osdmap e46: 8 total, 7 up, 8 in 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: from='osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:31.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:31 smithi114 ceph-mon[109024]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]: dispatch 2023-10-03T07:12:32.056 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:32.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:32 smithi031 ceph-mon[107170]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:32.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:32 smithi031 ceph-mon[107170]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:32.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:32 smithi031 ceph-mon[107170]: osdmap e47: 8 total, 7 up, 8 in 2023-10-03T07:12:32.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:32.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:32.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:32 smithi139 ceph-mon[109290]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:32.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:32 smithi139 ceph-mon[109290]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:32.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:32 smithi139 ceph-mon[109290]: osdmap e47: 8 total, 7 up, 8 in 2023-10-03T07:12:32.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:32.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:32.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:32 smithi114 ceph-mon[109024]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2023-10-03T07:12:32.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:32 smithi114 ceph-mon[109024]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi139", "root=default"]}]': finished 2023-10-03T07:12:32.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:32 smithi114 ceph-mon[109024]: osdmap e47: 8 total, 7 up, 8 in 2023-10-03T07:12:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:32.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:33.289 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:33.660 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":48,"num_osds":8,"num_up_osds":8,"osd_up_since":1696317152,"num_in_osds":8,"osd_in_since":1696317138,"num_remapped_pgs":0} 2023-10-03T07:12:33.661 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-03T07:12:33.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: purged_snaps scrub starts 2023-10-03T07:12:33.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: purged_snaps scrub ok 2023-10-03T07:12:33.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381] boot 2023-10-03T07:12:33.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: osdmap e48: 8 total, 8 up, 8 in 2023-10-03T07:12:33.736 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:33.737 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:33 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/4206674903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:33.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: purged_snaps scrub starts 2023-10-03T07:12:33.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: purged_snaps scrub ok 2023-10-03T07:12:33.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381] boot 2023-10-03T07:12:33.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: osdmap e48: 8 total, 8 up, 8 in 2023-10-03T07:12:33.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:33.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:33 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/4206674903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:33.951 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: purged_snaps scrub starts 2023-10-03T07:12:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: purged_snaps scrub ok 2023-10-03T07:12:33.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: osd.7 [v2:172.21.15.139:6816/4293964381,v1:172.21.15.139:6817/4293964381] boot 2023-10-03T07:12:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: osdmap e48: 8 total, 8 up, 8 in 2023-10-03T07:12:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2023-10-03T07:12:33.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:33 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/4206674903' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2023-10-03T07:12:34.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:34 smithi031 ceph-mon[107170]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:34.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:34 smithi031 ceph-mon[107170]: osdmap e49: 8 total, 8 up, 8 in 2023-10-03T07:12:34.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:34 smithi139 ceph-mon[109290]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:34.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:34 smithi139 ceph-mon[109290]: osdmap e49: 8 total, 8 up, 8 in 2023-10-03T07:12:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:34 smithi114 ceph-mon[109024]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:34.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:34 smithi114 ceph-mon[109024]: osdmap e49: 8 total, 8 up, 8 in 2023-10-03T07:12:35.038 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:35.038 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","created":"2023-10-03T07:08:40.883059+0000","modified":"2023-10-03T07:12:33.563495+0000","last_up_change":"2023-10-03T07:12:32.557228+0000","last_in_change":"2023-10-03T07:12:18.272632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T07:11:13.992578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32d40a81-22e7-4a48-94e9-7a836422ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6803","nonce":1219341756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6805","nonce":1219341756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6809","nonce":1219341756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6807","nonce":1219341756}]},"public_addr":"172.21.15.31:6803/1219341756","cluster_addr":"172.21.15.31:6805/1219341756","heartbeat_back_addr":"172.21.15.31:6809/1219341756","heartbeat_front_addr":"172.21.15.31:6807/1219341756","state":["exists","up"]},{"osd":1,"uuid":"32be39eb-7ab6-42a7-87f4-9e3a5b9684cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6811","nonce":2944524178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6813","nonce":2944524178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6817","nonce":2944524178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6815","nonce":2944524178}]},"public_addr":"172.21.15.31:6811/2944524178","cluster_addr":"172.21.15.31:6813/2944524178","heartbeat_back_addr":"172.21.15.31:6817/2944524178","heartbeat_front_addr":"172.21.15.31:6815/2944524178","state":["exists","up"]},{"osd":2,"uuid":"485d9558-1877-4502-9ae4-d2da6612140a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6801","nonce":1195543567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6803","nonce":1195543567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6807","nonce":1195543567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6805","nonce":1195543567}]},"public_addr":"172.21.15.114:6801/1195543567","cluster_addr":"172.21.15.114:6803/1195543567","heartbeat_back_addr":"172.21.15.114:6807/1195543567","heartbeat_front_addr":"172.21.15.114:6805/1195543567","state":["exists","up"]},{"osd":3,"uuid":"56329ffe-90ad-471a-9493-219054d65a29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6809","nonce":3292587599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6811","nonce":3292587599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6815","nonce":3292587599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6813","nonce":3292587599}]},"public_addr":"172.21.15.114:6809/3292587599","cluster_addr":"172.21.15.114:6811/3292587599","heartbeat_back_addr":"172.21.15.114:6815/3292587599","heartbeat_front_addr":"172.21.15.114:6813/3292587599","state":["exists","up"]},{"osd":4,"uuid":"c2014159-a7d2-4fd5-b06e-e08543bd98b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6817","nonce":1956269497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6819","nonce":1956269497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6823","nonce":1956269497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6821","nonce":1956269497}]},"public_addr":"172.21.15.114:6817/1956269497","cluster_addr":"172.21.15.114:6819/1956269497","heartbeat_back_addr":"172.21.15.114:6823/1956269497","heartbeat_front_addr":"172.21.15.114:6821/1956269497","state":["exists","up"]},{"osd":5,"uuid":"16d7dda5-89e5-440e-ba54-4ec47bd79cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6801","nonce":2746087010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6803","nonce":2746087010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6807","nonce":2746087010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6805","nonce":2746087010}]},"public_addr":"172.21.15.139:6801/2746087010","cluster_addr":"172.21.15.139:6803/2746087010","heartbeat_back_addr":"172.21.15.139:6807/2746087010","heartbeat_front_addr":"172.21.15.139:6805/2746087010","state":["exists","up"]},{"osd":6,"uuid":"9938d1fa-de80-46cc-a45a-fd4761b4ca49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6809","nonce":1462301427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6811","nonce":1462301427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6815","nonce":1462301427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6813","nonce":1462301427}]},"public_addr":"172.21.15.139:6809/1462301427","cluster_addr":"172.21.15.139:6811/1462301427","heartbeat_back_addr":"172.21.15.139:6815/1462301427","heartbeat_front_addr":"172.21.15.139:6813/1462301427","state":["exists","up"]},{"osd":7,"uuid":"20b1bc93-429a-4914-842f-243ac5be829a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6817","nonce":4293964381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6819","nonce":4293964381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6823","nonce":4293964381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6821","nonce":4293964381}]},"public_addr":"172.21.15.139:6817/4293964381","cluster_addr":"172.21.15.139:6819/4293964381","heartbeat_back_addr":"172.21.15.139:6823/4293964381","heartbeat_front_addr":"172.21.15.139:6821/4293964381","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:38.745841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:55.832062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:11.529540+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:27.384893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:43.447915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:59.004756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:15.075164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:30.701395+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2357718878":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2835788381":"2023-10-04T07:09:29.446440+0000","172.21.15.31:6801/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/3273043545":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6800/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6801/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/1455854068":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/2884724052":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/4242800476":"2023-10-04T07:09:29.446440+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T07:12:35.409 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2023-10-03T07:11:13.992578+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2023-10-03T07:12:35.410 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd pool get .mgr pg_num 2023-10-03T07:12:35.703 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:35.812 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:35 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1607286560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:35.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:35 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1607286560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:35.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:35 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1607286560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:36.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:36 smithi031 ceph-mon[107170]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:36.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:36 smithi139 ceph-mon[109290]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:36.935 INFO:teuthology.orchestra.run.smithi031.stdout:pg_num: 1 2023-10-03T07:12:36.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:36 smithi114 ceph-mon[109024]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:37.266 INFO:tasks.cephadm:Setting up client nodes... 2023-10-03T07:12:37.266 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-03T07:12:37.511 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:37.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:37 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1823000956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-03T07:12:37.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:37 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1823000956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-03T07:12:38.001 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:37 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1823000956' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2023-10-03T07:12:38.707 INFO:teuthology.orchestra.run.smithi031.stdout:[client.0] 2023-10-03T07:12:38.707 INFO:teuthology.orchestra.run.smithi031.stdout: key = AQDmvhtlFF30KRAAKyu/v5c79lLpByUbwQLE0A== 2023-10-03T07:12:38.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:38 smithi139 ceph-mon[109290]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:38.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:38 smithi114 ceph-mon[109024]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:39.034 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:38 smithi031 ceph-mon[107170]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:39.100 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:12:39.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2023-10-03T07:12:39.101 DEBUG:teuthology.orchestra.run.smithi031:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2023-10-03T07:12:39.151 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-03T07:12:39.381 INFO:teuthology.orchestra.run.smithi114.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.b/config 2023-10-03T07:12:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:39 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:39.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:39 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:39.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:39 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:39.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:39 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:40.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:39 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:40.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:39 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3457127088' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:40.551 INFO:teuthology.orchestra.run.smithi114.stdout:[client.1] 2023-10-03T07:12:40.551 INFO:teuthology.orchestra.run.smithi114.stdout: key = AQDovhtlW9qkIBAAMsQ0jTuAQyUwNccvJ1LHrA== 2023-10-03T07:12:40.724 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:40 smithi114 ceph-mon[109024]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:40.724 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:40 smithi114 ceph-mon[109024]: from='client.? 172.21.15.114:0/3294903843' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:40.724 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:40 smithi114 ceph-mon[109024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:40.724 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:40 smithi114 ceph-mon[109024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:40.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:40 smithi139 ceph-mon[109290]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:40.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:40 smithi139 ceph-mon[109290]: from='client.? 172.21.15.114:0/3294903843' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:40.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:40 smithi139 ceph-mon[109290]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:40.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:40 smithi139 ceph-mon[109290]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:40.927 DEBUG:teuthology.orchestra.run.smithi114:> set -ex 2023-10-03T07:12:40.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2023-10-03T07:12:40.928 DEBUG:teuthology.orchestra.run.smithi114:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2023-10-03T07:12:40.969 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2023-10-03T07:12:41.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:40 smithi031 ceph-mon[107170]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:41.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:40 smithi031 ceph-mon[107170]: from='client.? 172.21.15.114:0/3294903843' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:41.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:40 smithi031 ceph-mon[107170]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:41.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:40 smithi031 ceph-mon[107170]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:41.201 INFO:teuthology.orchestra.run.smithi139.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.c/config 2023-10-03T07:12:42.340 INFO:teuthology.orchestra.run.smithi139.stdout:[client.2] 2023-10-03T07:12:42.341 INFO:teuthology.orchestra.run.smithi139.stdout: key = AQDqvhtlOUUdFBAAQpzrWHdZGyZM2OzlNBYlPQ== 2023-10-03T07:12:42.722 DEBUG:teuthology.orchestra.run.smithi139:> set -ex 2023-10-03T07:12:42.722 DEBUG:teuthology.orchestra.run.smithi139:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2023-10-03T07:12:42.722 DEBUG:teuthology.orchestra.run.smithi139:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2023-10-03T07:12:42.763 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2023-10-03T07:12:42.764 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2023-10-03T07:12:42.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph mgr dump --format=json 2023-10-03T07:12:42.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:42 smithi139 ceph-mon[109290]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:42.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:42 smithi139 ceph-mon[109290]: from='client.? 172.21.15.139:0/605486956' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:42.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:42 smithi139 ceph-mon[109290]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:42.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:42 smithi139 ceph-mon[109290]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:42 smithi114 ceph-mon[109024]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:42 smithi114 ceph-mon[109024]: from='client.? 172.21.15.139:0/605486956' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:42 smithi114 ceph-mon[109024]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:42.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:42 smithi114 ceph-mon[109024]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:43.011 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:43.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:42 smithi031 ceph-mon[107170]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:42 smithi031 ceph-mon[107170]: from='client.? 172.21.15.139:0/605486956' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:42 smithi031 ceph-mon[107170]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2023-10-03T07:12:43.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:42 smithi031 ceph-mon[107170]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2023-10-03T07:12:44.253 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:44.633 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":14,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6800","nonce":384413686},{"type":"v1","addr":"172.21.15.31:6801","nonce":384413686}]},"active_addr":"172.21.15.31:6801/384413686","active_change":"2023-10-03T07:09:29.446572+0000","active_mgr_features":4540138322906710015,"available":true,"standbys":[],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:0.0.3","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"secs","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"noautoscale":{"name":"noautoscale","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"global autoscale flag","long_desc":"Option to turn on/off the autoscaler for all pools","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.31:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":2951131772}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":114831526}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":865386569}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.31:0","nonce":77413588}]}]} 2023-10-03T07:12:44.637 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2023-10-03T07:12:44.637 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2023-10-03T07:12:44.637 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-03T07:12:44.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:44 smithi031 ceph-mon[107170]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:44.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:44 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3665388143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-03T07:12:44.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:44 smithi139 ceph-mon[109290]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:44.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:44 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3665388143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-03T07:12:44.886 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:44 smithi114 ceph-mon[109024]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:44.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:44 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3665388143' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2023-10-03T07:12:46.057 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:46.057 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","created":"2023-10-03T07:08:40.883059+0000","modified":"2023-10-03T07:12:33.563495+0000","last_up_change":"2023-10-03T07:12:32.557228+0000","last_in_change":"2023-10-03T07:12:18.272632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T07:11:13.992578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32d40a81-22e7-4a48-94e9-7a836422ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6803","nonce":1219341756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6805","nonce":1219341756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6809","nonce":1219341756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6807","nonce":1219341756}]},"public_addr":"172.21.15.31:6803/1219341756","cluster_addr":"172.21.15.31:6805/1219341756","heartbeat_back_addr":"172.21.15.31:6809/1219341756","heartbeat_front_addr":"172.21.15.31:6807/1219341756","state":["exists","up"]},{"osd":1,"uuid":"32be39eb-7ab6-42a7-87f4-9e3a5b9684cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6811","nonce":2944524178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6813","nonce":2944524178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6817","nonce":2944524178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6815","nonce":2944524178}]},"public_addr":"172.21.15.31:6811/2944524178","cluster_addr":"172.21.15.31:6813/2944524178","heartbeat_back_addr":"172.21.15.31:6817/2944524178","heartbeat_front_addr":"172.21.15.31:6815/2944524178","state":["exists","up"]},{"osd":2,"uuid":"485d9558-1877-4502-9ae4-d2da6612140a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6801","nonce":1195543567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6803","nonce":1195543567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6807","nonce":1195543567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6805","nonce":1195543567}]},"public_addr":"172.21.15.114:6801/1195543567","cluster_addr":"172.21.15.114:6803/1195543567","heartbeat_back_addr":"172.21.15.114:6807/1195543567","heartbeat_front_addr":"172.21.15.114:6805/1195543567","state":["exists","up"]},{"osd":3,"uuid":"56329ffe-90ad-471a-9493-219054d65a29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6809","nonce":3292587599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6811","nonce":3292587599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6815","nonce":3292587599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6813","nonce":3292587599}]},"public_addr":"172.21.15.114:6809/3292587599","cluster_addr":"172.21.15.114:6811/3292587599","heartbeat_back_addr":"172.21.15.114:6815/3292587599","heartbeat_front_addr":"172.21.15.114:6813/3292587599","state":["exists","up"]},{"osd":4,"uuid":"c2014159-a7d2-4fd5-b06e-e08543bd98b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6817","nonce":1956269497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6819","nonce":1956269497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6823","nonce":1956269497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6821","nonce":1956269497}]},"public_addr":"172.21.15.114:6817/1956269497","cluster_addr":"172.21.15.114:6819/1956269497","heartbeat_back_addr":"172.21.15.114:6823/1956269497","heartbeat_front_addr":"172.21.15.114:6821/1956269497","state":["exists","up"]},{"osd":5,"uuid":"16d7dda5-89e5-440e-ba54-4ec47bd79cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6801","nonce":2746087010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6803","nonce":2746087010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6807","nonce":2746087010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6805","nonce":2746087010}]},"public_addr":"172.21.15.139:6801/2746087010","cluster_addr":"172.21.15.139:6803/2746087010","heartbeat_back_addr":"172.21.15.139:6807/2746087010","heartbeat_front_addr":"172.21.15.139:6805/2746087010","state":["exists","up"]},{"osd":6,"uuid":"9938d1fa-de80-46cc-a45a-fd4761b4ca49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6809","nonce":1462301427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6811","nonce":1462301427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6815","nonce":1462301427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6813","nonce":1462301427}]},"public_addr":"172.21.15.139:6809/1462301427","cluster_addr":"172.21.15.139:6811/1462301427","heartbeat_back_addr":"172.21.15.139:6815/1462301427","heartbeat_front_addr":"172.21.15.139:6813/1462301427","state":["exists","up"]},{"osd":7,"uuid":"20b1bc93-429a-4914-842f-243ac5be829a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6817","nonce":4293964381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6819","nonce":4293964381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6823","nonce":4293964381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6821","nonce":4293964381}]},"public_addr":"172.21.15.139:6817/4293964381","cluster_addr":"172.21.15.139:6819/4293964381","heartbeat_back_addr":"172.21.15.139:6823/4293964381","heartbeat_front_addr":"172.21.15.139:6821/4293964381","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:38.745841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:55.832062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:11.529540+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:27.384893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:43.447915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:59.004756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:15.075164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:30.701395+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2357718878":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2835788381":"2023-10-04T07:09:29.446440+0000","172.21.15.31:6801/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/3273043545":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6800/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6801/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/1455854068":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/2884724052":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/4242800476":"2023-10-04T07:09:29.446440+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T07:12:46.442 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2023-10-03T07:12:46.442 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd dump --format=json 2023-10-03T07:12:46.733 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:46 smithi031 ceph-mon[107170]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:46.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:46 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3599925621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:46.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:46 smithi139 ceph-mon[109290]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:46.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:46 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3599925621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:46.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:46 smithi114 ceph-mon[109024]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:46.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:46 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3599925621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:47.880 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:12:47.881 INFO:teuthology.orchestra.run.smithi031.stdout:{"epoch":49,"fsid":"8ad20b58-61bb-11ee-8db5-212e2dc638e7","created":"2023-10-03T07:08:40.883059+0000","modified":"2023-10-03T07:12:33.563495+0000","last_up_change":"2023-10-03T07:12:32.557228+0000","last_in_change":"2023-10-03T07:12:18.272632+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"reef","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2023-10-03T07:11:13.992578+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"32d40a81-22e7-4a48-94e9-7a836422ea3b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6802","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6803","nonce":1219341756}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6804","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6805","nonce":1219341756}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6808","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6809","nonce":1219341756}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6806","nonce":1219341756},{"type":"v1","addr":"172.21.15.31:6807","nonce":1219341756}]},"public_addr":"172.21.15.31:6803/1219341756","cluster_addr":"172.21.15.31:6805/1219341756","heartbeat_back_addr":"172.21.15.31:6809/1219341756","heartbeat_front_addr":"172.21.15.31:6807/1219341756","state":["exists","up"]},{"osd":1,"uuid":"32be39eb-7ab6-42a7-87f4-9e3a5b9684cf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6810","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6811","nonce":2944524178}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6812","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6813","nonce":2944524178}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6816","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6817","nonce":2944524178}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.31:6814","nonce":2944524178},{"type":"v1","addr":"172.21.15.31:6815","nonce":2944524178}]},"public_addr":"172.21.15.31:6811/2944524178","cluster_addr":"172.21.15.31:6813/2944524178","heartbeat_back_addr":"172.21.15.31:6817/2944524178","heartbeat_front_addr":"172.21.15.31:6815/2944524178","state":["exists","up"]},{"osd":2,"uuid":"485d9558-1877-4502-9ae4-d2da6612140a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6800","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6801","nonce":1195543567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6802","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6803","nonce":1195543567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6806","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6807","nonce":1195543567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6804","nonce":1195543567},{"type":"v1","addr":"172.21.15.114:6805","nonce":1195543567}]},"public_addr":"172.21.15.114:6801/1195543567","cluster_addr":"172.21.15.114:6803/1195543567","heartbeat_back_addr":"172.21.15.114:6807/1195543567","heartbeat_front_addr":"172.21.15.114:6805/1195543567","state":["exists","up"]},{"osd":3,"uuid":"56329ffe-90ad-471a-9493-219054d65a29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":24,"up_thru":39,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6808","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6809","nonce":3292587599}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6810","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6811","nonce":3292587599}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6814","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6815","nonce":3292587599}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6812","nonce":3292587599},{"type":"v1","addr":"172.21.15.114:6813","nonce":3292587599}]},"public_addr":"172.21.15.114:6809/3292587599","cluster_addr":"172.21.15.114:6811/3292587599","heartbeat_back_addr":"172.21.15.114:6815/3292587599","heartbeat_front_addr":"172.21.15.114:6813/3292587599","state":["exists","up"]},{"osd":4,"uuid":"c2014159-a7d2-4fd5-b06e-e08543bd98b8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6816","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6817","nonce":1956269497}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6818","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6819","nonce":1956269497}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6822","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6823","nonce":1956269497}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.114:6820","nonce":1956269497},{"type":"v1","addr":"172.21.15.114:6821","nonce":1956269497}]},"public_addr":"172.21.15.114:6817/1956269497","cluster_addr":"172.21.15.114:6819/1956269497","heartbeat_back_addr":"172.21.15.114:6823/1956269497","heartbeat_front_addr":"172.21.15.114:6821/1956269497","state":["exists","up"]},{"osd":5,"uuid":"16d7dda5-89e5-440e-ba54-4ec47bd79cb2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6800","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6801","nonce":2746087010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6802","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6803","nonce":2746087010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6806","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6807","nonce":2746087010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6804","nonce":2746087010},{"type":"v1","addr":"172.21.15.139:6805","nonce":2746087010}]},"public_addr":"172.21.15.139:6801/2746087010","cluster_addr":"172.21.15.139:6803/2746087010","heartbeat_back_addr":"172.21.15.139:6807/2746087010","heartbeat_front_addr":"172.21.15.139:6805/2746087010","state":["exists","up"]},{"osd":6,"uuid":"9938d1fa-de80-46cc-a45a-fd4761b4ca49","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":43,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6808","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6809","nonce":1462301427}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6810","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6811","nonce":1462301427}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6814","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6815","nonce":1462301427}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6812","nonce":1462301427},{"type":"v1","addr":"172.21.15.139:6813","nonce":1462301427}]},"public_addr":"172.21.15.139:6809/1462301427","cluster_addr":"172.21.15.139:6811/1462301427","heartbeat_back_addr":"172.21.15.139:6815/1462301427","heartbeat_front_addr":"172.21.15.139:6813/1462301427","state":["exists","up"]},{"osd":7,"uuid":"20b1bc93-429a-4914-842f-243ac5be829a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6816","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6817","nonce":4293964381}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6818","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6819","nonce":4293964381}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6822","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6823","nonce":4293964381}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.139:6820","nonce":4293964381},{"type":"v1","addr":"172.21.15.139:6821","nonce":4293964381}]},"public_addr":"172.21.15.139:6817/4293964381","cluster_addr":"172.21.15.139:6819/4293964381","heartbeat_back_addr":"172.21.15.139:6823/4293964381","heartbeat_front_addr":"172.21.15.139:6821/4293964381","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:38.745841+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:10:55.832062+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:11.529540+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:27.384893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:43.447915+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:11:59.004756+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:15.075164+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138322906710015,"old_weight":0,"last_purged_snaps_scrub":"2023-10-03T07:12:30.701395+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.31:6800/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2357718878":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/2835788381":"2023-10-04T07:09:29.446440+0000","172.21.15.31:6801/1795458781":"2023-10-04T07:09:29.446440+0000","172.21.15.31:0/3273043545":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6800/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:6801/3440384701":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/1455854068":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/2884724052":"2023-10-04T07:09:04.952213+0000","172.21.15.31:0/4242800476":"2023-10-04T07:09:29.446440+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2023-10-03T07:12:48.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.0 flush_pg_stats 2023-10-03T07:12:48.259 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.1 flush_pg_stats 2023-10-03T07:12:48.260 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.2 flush_pg_stats 2023-10-03T07:12:48.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.3 flush_pg_stats 2023-10-03T07:12:48.261 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.4 flush_pg_stats 2023-10-03T07:12:48.262 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.5 flush_pg_stats 2023-10-03T07:12:48.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.6 flush_pg_stats 2023-10-03T07:12:48.263 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph tell osd.7 flush_pg_stats 2023-10-03T07:12:48.731 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:48.747 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:48.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:48 smithi031 ceph-mon[107170]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:48.869 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:48 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/2231044688' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:48.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:48 smithi139 ceph-mon[109290]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:48.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:48 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/2231044688' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:48.977 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:48.978 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:48 smithi114 ceph-mon[109024]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:48.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:48 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/2231044688' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2023-10-03T07:12:49.357 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:50.216 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:50.703 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:50 smithi031 ceph-mon[107170]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:50.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:50 smithi114 ceph-mon[109024]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:50.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:50 smithi139 ceph-mon[109290]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:52.254 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:52.424 INFO:teuthology.orchestra.run.smithi031.stdout:34359738397 2023-10-03T07:12:52.424 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.0 2023-10-03T07:12:52.682 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:52 smithi031 ceph-mon[107170]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:52.706 INFO:teuthology.orchestra.run.smithi031.stdout:55834574874 2023-10-03T07:12:52.706 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.1 2023-10-03T07:12:52.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:52 smithi139 ceph-mon[109290]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:52.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:52 smithi114 ceph-mon[109024]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:53.733 INFO:teuthology.orchestra.run.smithi031.stdout:154618822670 2023-10-03T07:12:53.734 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.5 2023-10-03T07:12:53.961 INFO:teuthology.orchestra.run.smithi031.stdout:137438953489 2023-10-03T07:12:53.962 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.4 2023-10-03T07:12:54.227 INFO:teuthology.orchestra.run.smithi031.stdout:206158430215 2023-10-03T07:12:54.228 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.7 2023-10-03T07:12:54.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:54 smithi031 ceph-mon[107170]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:54.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:54 smithi139 ceph-mon[109290]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:54.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:54 smithi114 ceph-mon[109024]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:55.246 INFO:teuthology.orchestra.run.smithi031.stdout:103079215124 2023-10-03T07:12:55.246 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.3 2023-10-03T07:12:55.394 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:55.732 INFO:teuthology.orchestra.run.smithi031.stdout:77309411350 2023-10-03T07:12:55.732 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.2 2023-10-03T07:12:55.797 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:55.948 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:56.288 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:56.618 INFO:teuthology.orchestra.run.smithi031.stdout:184683593739 2023-10-03T07:12:56.618 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph osd last-stat-seq osd.6 2023-10-03T07:12:56.771 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:56 smithi031 ceph-mon[107170]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:56.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:56 smithi139 ceph-mon[109290]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:56.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:56 smithi114 ceph-mon[109024]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:57.374 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:57.870 INFO:teuthology.orchestra.run.smithi031.stdout:154618822671 2023-10-03T07:12:57.907 INFO:teuthology.orchestra.run.smithi031.stdout:55834574875 2023-10-03T07:12:58.440 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:58.440 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:12:58.459 INFO:teuthology.orchestra.run.smithi031.stdout:137438953490 2023-10-03T07:12:58.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:58 smithi031 ceph-mon[107170]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:58.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:58 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/2655448696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-03T07:12:58.635 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:58 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1658155090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-03T07:12:58.636 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:12:58 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/2967088478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-03T07:12:58.782 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822670 got 154618822671 for osd.5 2023-10-03T07:12:58.783 DEBUG:teuthology.parallel:result is None 2023-10-03T07:12:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:58 smithi139 ceph-mon[109290]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:58 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/2655448696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-03T07:12:58.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:58 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1658155090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-03T07:12:58.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:12:58 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/2967088478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-03T07:12:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:58 smithi114 ceph-mon[109024]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:12:58.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:58 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/2655448696' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2023-10-03T07:12:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:58 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1658155090' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2023-10-03T07:12:58.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:12:58 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/2967088478' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2023-10-03T07:12:59.553 INFO:teuthology.orchestra.run.smithi031.stdout:34359738398 2023-10-03T07:12:59.833 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574874 got 55834574875 for osd.1 2023-10-03T07:12:59.833 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:00.122 INFO:teuthology.orchestra.run.smithi031.stdout:206158430215 2023-10-03T07:13:00.139 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953489 got 137438953490 for osd.4 2023-10-03T07:13:00.139 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:00.739 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:00 smithi031 ceph-mon[107170]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:00.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:00 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1132120865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-03T07:13:00.740 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:00 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3659528558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-03T07:13:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:00 smithi114 ceph-mon[109024]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:00 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1132120865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-03T07:13:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:00 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3659528558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-03T07:13:00.867 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430215 got 206158430215 for osd.7 2023-10-03T07:13:00.867 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:00 smithi139 ceph-mon[109290]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:00 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1132120865' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2023-10-03T07:13:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:00 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3659528558' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2023-10-03T07:13:01.130 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:01.309 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738397 got 34359738398 for osd.0 2023-10-03T07:13:01.309 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:01.317 INFO:teuthology.orchestra.run.smithi031.stdout:77309411351 2023-10-03T07:13:01.351 INFO:teuthology.orchestra.run.smithi031.stdout:103079215125 2023-10-03T07:13:01.627 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:01 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/870124266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-03T07:13:01.628 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:01 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1250466939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-03T07:13:01.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:01 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/870124266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-03T07:13:01.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:01 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1250466939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-03T07:13:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:01 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/870124266' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2023-10-03T07:13:01.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:01 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1250466939' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2023-10-03T07:13:02.145 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411350 got 77309411351 for osd.2 2023-10-03T07:13:02.145 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:02.324 INFO:tasks.cephadm.ceph_manager.ceph:need seq 103079215124 got 103079215125 for osd.3 2023-10-03T07:13:02.324 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:02.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:02 smithi031 ceph-mon[107170]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:02.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:02 smithi114 ceph-mon[109024]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:02.820 INFO:teuthology.orchestra.run.smithi031.stdout:184683593740 2023-10-03T07:13:02.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:02 smithi139 ceph-mon[109290]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:03.205 INFO:tasks.cephadm.ceph_manager.ceph:need seq 184683593739 got 184683593740 for osd.6 2023-10-03T07:13:03.205 DEBUG:teuthology.parallel:result is None 2023-10-03T07:13:03.205 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2023-10-03T07:13:03.205 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph pg dump --format=json 2023-10-03T07:13:03.456 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:03.671 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:03 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/529058807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-03T07:13:03.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:03 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/529058807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-03T07:13:03.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:03 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/529058807' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2023-10-03T07:13:04.566 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:04 smithi031 ceph-mon[107170]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:04.624 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:04.626 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T07:13:04.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:04 smithi114 ceph-mon[109024]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:04.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:04 smithi139 ceph-mon[109290]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:05.047 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":146,"stamp":"2023-10-03T07:13:03.491007+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":221600,"kb_used_data":5412,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":749739616,"statfs":{"total":767960285184,"available":767733366784,"internally_reserved":0,"allocated":5541888,"data_stored":3303180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002372"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":238,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T07:12:32.656224+0000","last_change":"2023-10-03T07:12:04.318463+0000","last_active":"2023-10-03T07:12:32.656224+0000","last_peered":"2023-10-03T07:12:32.656224+0000","last_clean":"2023-10-03T07:12:32.656224+0000","last_became_active":"2023-10-03T07:12:04.317990+0000","last_became_peered":"2023-10-03T07:12:04.317990+0000","last_unstale":"2023-10-03T07:12:32.656224+0000","last_undegraded":"2023-10-03T07:12:32.656224+0000","last_fullsized":"2023-10-03T07:12:32.656224+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T07:11:14.448226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T07:11:14.448226+0000","last_clean_scrub_stamp":"2023-10-03T07:11:14.448226+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:40:51.745591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:13:02 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.35599999999999998,"15min":0.35099999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Oct 3 07:12:15 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Tue Oct 3 07:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.753},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80300000000000005}]},{"osd":4,"last update":"Tue Oct 3 07:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.70299999999999996}]},{"osd":5,"last update":"Tue Oct 3 07:13:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.871}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 07:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Oct 3 07:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Tue Oct 3 07:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.496}]},{"osd":4,"last update":"Tue Oct 3 07:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.1429999999999998,"5min":6.1429999999999998,"15min":6.1429999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":19.925000000000001,"5min":19.925000000000001,"15min":19.925000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":781680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42699999999999999}]},{"osd":1,"last update":"Tue Oct 3 07:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.502},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Oct 3 07:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Oct 3 07:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.44800000000000001}]},{"osd":5,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":3,"up_from":24,"seq":103079215126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":781953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.496},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Oct 3 07:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56799999999999995}]},{"osd":5,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]},{"osd":4,"up_from":32,"seq":137438953491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.498},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52800000000000002}]},{"osd":2,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.443}]},{"osd":3,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":5,"up_from":36,"seq":154618822672,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1060864,"data_stored":781622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68899999999999995}]},{"osd":1,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.61499999999999999}]},{"osd":2,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.58099999999999996}]},{"osd":3,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64300000000000002}]},{"osd":4,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T07:13:05.049 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph pg dump --format=json 2023-10-03T07:13:05.299 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:05.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:05 smithi114 ceph-mon[109024]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:05.788 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:05 smithi031 ceph-mon[107170]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:05.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:05 smithi139 ceph-mon[109290]: from='client.14499 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:06.418 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:06.420 INFO:teuthology.orchestra.run.smithi031.stderr:dumped all 2023-10-03T07:13:06.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:06 smithi114 ceph-mon[109024]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:06.746 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:06 smithi031 ceph-mon[107170]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:06.805 INFO:teuthology.orchestra.run.smithi031.stdout:{"pg_ready":true,"pg_map":{"version":147,"stamp":"2023-10-03T07:13:05.491414+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":3,"kb":749961216,"kb_used":221600,"kb_used_data":5412,"kb_used_omap":13,"kb_used_meta":216050,"kb_avail":749739616,"statfs":{"total":767960285184,"available":767733366784,"internally_reserved":0,"allocated":5541888,"data_stored":3303180,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":14248,"internal_metadata":221235288},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002410"},"pg_stats":[{"pgid":"1.0","version":"21'137","reported_seq":238,"reported_epoch":48,"state":"active+clean","last_fresh":"2023-10-03T07:12:32.656224+0000","last_change":"2023-10-03T07:12:04.318463+0000","last_active":"2023-10-03T07:12:32.656224+0000","last_peered":"2023-10-03T07:12:32.656224+0000","last_clean":"2023-10-03T07:12:32.656224+0000","last_became_active":"2023-10-03T07:12:04.317990+0000","last_became_peered":"2023-10-03T07:12:04.317990+0000","last_unstale":"2023-10-03T07:12:32.656224+0000","last_undegraded":"2023-10-03T07:12:32.656224+0000","last_fullsized":"2023-10-03T07:12:32.656224+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2023-10-03T07:11:14.448226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2023-10-03T07:11:14.448226+0000","last_clean_scrub_stamp":"2023-10-03T07:11:14.448226+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2023-10-04T17:40:51.745591+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":48,"seq":206158430217,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56200000000000006}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63900000000000001}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81299999999999994}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68000000000000005}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72299999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70599999999999996}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66000000000000003}]}]},{"osd":6,"up_from":43,"seq":184683593741,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52200000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53400000000000003}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64200000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.751}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.73399999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57799999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]}]},{"osd":1,"up_from":13,"seq":55834574877,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:13:02 2023","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.33900000000000002,"15min":0.33300000000000002},"min":{"1min":0.26200000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.38500000000000001,"5min":0.35599999999999998,"15min":0.35099999999999998},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.60099999999999998}]},{"osd":2,"last update":"Tue Oct 3 07:12:15 2023","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.505},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.79100000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.72299999999999998}]},{"osd":3,"last update":"Tue Oct 3 07:12:28 2023","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81000000000000005,"5min":0.81000000000000005,"15min":0.81000000000000005},"last":0.753},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82699999999999996,"5min":0.82699999999999996,"15min":0.82699999999999996},"last":0.80300000000000005}]},{"osd":4,"last update":"Tue Oct 3 07:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":0.84499999999999997},"last":0.76900000000000002},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.70299999999999996}]},{"osd":5,"last update":"Tue Oct 3 07:13:02 2023","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.95699999999999996,"5min":0.95699999999999996,"15min":0.95699999999999996},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.64600000000000002,"15min":0.64600000000000002},"min":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.871}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97899999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.93799999999999994}]}]},{"osd":0,"up_from":8,"seq":34359738399,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27484,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717668,"statfs":{"total":95995035648,"available":95966892032,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Tue Oct 3 07:11:59 2023","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.53900000000000003,"5min":0.53900000000000003,"15min":0.53900000000000003},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38,"15min":0.38},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.46100000000000002}]},{"osd":2,"last update":"Tue Oct 3 07:12:13 2023","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.59399999999999997},"min":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.57099999999999995}]},{"osd":3,"last update":"Tue Oct 3 07:12:31 2023","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.71399999999999997,"5min":0.71399999999999997,"15min":0.71399999999999997},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.55400000000000005,"15min":0.55400000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.496}]},{"osd":4,"last update":"Tue Oct 3 07:12:48 2023","interfaces":[{"interface":"back","average":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"min":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"max":{"1min":6.1429999999999998,"5min":6.1429999999999998,"15min":6.1429999999999998},"last":0.53300000000000003},{"interface":"front","average":{"1min":1.5760000000000001,"5min":1.5760000000000001,"15min":1.5760000000000001},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":19.925000000000001,"5min":19.925000000000001,"15min":19.925000000000001},"last":0.51800000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.71899999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.66700000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.70499999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411352,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1060864,"data_stored":781680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.86099999999999999,"5min":0.86099999999999999,"15min":0.86099999999999999},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.77600000000000002,"5min":0.77600000000000002,"15min":0.77600000000000002},"last":0.42699999999999999}]},{"osd":1,"last update":"Tue Oct 3 07:12:14 2023","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.502},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54900000000000004},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Tue Oct 3 07:12:29 2023","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"last":0.47399999999999998}]},{"osd":4,"last update":"Tue Oct 3 07:12:46 2023","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.45400000000000001,"15min":0.45400000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.63100000000000001,"5min":0.63100000000000001,"15min":0.63100000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.66100000000000003,"5min":0.66100000000000003,"15min":0.66100000000000003},"last":0.44800000000000001}]},{"osd":5,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.87,"5min":0.87,"15min":0.87},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.55200000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67000000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.65400000000000003}]}]},{"osd":3,"up_from":24,"seq":103079215126,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1040,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1064960,"data_stored":781953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.496},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53600000000000003,"15min":0.53600000000000003},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.45000000000000001}]},{"osd":1,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.71799999999999997,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.78200000000000003},"last":0.53500000000000003}]},{"osd":2,"last update":"Tue Oct 3 07:12:30 2023","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.434,"5min":0.434,"15min":0.434},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.56899999999999995},"last":0.59199999999999997}]},{"osd":4,"last update":"Tue Oct 3 07:12:45 2023","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.432,"5min":0.432,"15min":0.432},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.60199999999999998,"5min":0.60199999999999998,"15min":0.60199999999999998},"last":0.56799999999999995}]},{"osd":5,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60999999999999999,"15min":0.60999999999999999},"min":{"1min":0.442,"5min":0.442,"15min":0.442},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.85299999999999998},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.61199999999999999},"min":{"1min":0.46700000000000003,"5min":0.46700000000000003,"15min":0.46700000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.71099999999999997}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72399999999999998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]}]},{"osd":4,"up_from":32,"seq":137438953491,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27480,"kb_used_data":460,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717672,"statfs":{"total":95995035648,"available":95966896128,"internally_reserved":0,"allocated":471040,"data_stored":191585,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.53100000000000003},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.78100000000000003,"5min":0.78100000000000003,"15min":0.78100000000000003},"last":0.498},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.53700000000000003,"15min":0.53700000000000003},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.81999999999999995,"5min":0.81999999999999995,"15min":0.81999999999999995},"last":0.51300000000000001}]},{"osd":1,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.39700000000000002,"5min":0.39700000000000002,"15min":0.39700000000000002},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.52800000000000002}]},{"osd":2,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.60499999999999998},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"last":0.443}]},{"osd":3,"last update":"Tue Oct 3 07:12:49 2023","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.34699999999999998},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.41799999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64100000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.58299999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":5,"up_from":36,"seq":154618822672,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1060864,"data_stored":781622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1781,"internal_metadata":27654411},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.94299999999999995,"5min":0.94299999999999995,"15min":0.94299999999999995},"last":0.68899999999999995}]},{"osd":1,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.88500000000000001,"5min":0.88500000000000001,"15min":0.88500000000000001},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.61499999999999999}]},{"osd":2,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.41999999999999998},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.64400000000000002,"5min":0.64400000000000002,"15min":0.64400000000000002},"min":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"max":{"1min":0.878,"5min":0.878,"15min":0.878},"last":0.58099999999999996}]},{"osd":3,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":0.80300000000000005},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"min":{"1min":0.375,"5min":0.375,"15min":0.375},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.64300000000000002}]},{"osd":4,"last update":"Tue Oct 3 07:13:01 2023","interfaces":[{"interface":"back","average":{"1min":0.65800000000000003,"5min":0.65800000000000003,"15min":0.65800000000000003},"min":{"1min":0.436,"5min":0.436,"15min":0.436},"max":{"1min":0.89400000000000002,"5min":0.89400000000000002,"15min":0.89400000000000002},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"min":{"1min":0.46000000000000002,"5min":0.46000000000000002,"15min":0.46000000000000002},"max":{"1min":0.875,"5min":0.875,"15min":0.875},"last":0.63}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38700000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2023-10-03T07:13:06.807 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2023-10-03T07:13:06.807 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2023-10-03T07:13:06.808 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2023-10-03T07:13:06.808 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- ceph health --format=json 2023-10-03T07:13:06.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:06 smithi139 ceph-mon[109290]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:07.055 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:07.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:07 smithi031 ceph-mon[107170]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:07.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:07 smithi114 ceph-mon[109024]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:07.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:07 smithi139 ceph-mon[109290]: from='client.14505 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-10-03T07:13:08.288 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:08.288 INFO:teuthology.orchestra.run.smithi031.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2023-10-03T07:13:08.679 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2023-10-03T07:13:08.679 INFO:tasks.cephadm:Setup complete, yielding 2023-10-03T07:13:08.679 INFO:teuthology.run_tasks:Running task cephadm.shell... 2023-10-03T07:13:08.689 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T07:13:08.690 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch status' 2023-10-03T07:13:08.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:08 smithi031 ceph-mon[107170]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:08.723 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:08 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/811783274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-03T07:13:08.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:08 smithi114 ceph-mon[109024]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:08.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:08 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/811783274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-03T07:13:08.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:08 smithi139 ceph-mon[109290]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:08.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:08 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/811783274' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2023-10-03T07:13:08.938 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:10.194 INFO:teuthology.orchestra.run.smithi031.stdout:Backend: cephadm 2023-10-03T07:13:10.194 INFO:teuthology.orchestra.run.smithi031.stdout:Available: Yes 2023-10-03T07:13:10.194 INFO:teuthology.orchestra.run.smithi031.stdout:Paused: No 2023-10-03T07:13:10.487 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:10 smithi031 ceph-mon[107170]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:10.571 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ps' 2023-10-03T07:13:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:10 smithi114 ceph-mon[109024]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:10.867 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:10.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:10 smithi139 ceph-mon[109290]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:11.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:11 smithi031 ceph-mon[107170]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:11.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:11 smithi114 ceph-mon[109024]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:11.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:11 smithi139 ceph-mon[109290]: from='client.14517 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:12.031 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-03T07:13:12.031 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (4m) - 4m 489M - 18.0.0-5817-gbcea8c48 fa922c444906 2e68f5c77307 2023-10-03T07:13:12.031 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (4m) - 4m 47.6M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 1b0e77539eb0 2023-10-03T07:13:12.031 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi114 running (3m) - 3m 43.4M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 97e2dcb1ebad 2023-10-03T07:13:12.031 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi139 running (3m) - 3m 47.2M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 4fb98de11a35 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (2m) - 2m 29.0M 8559M 18.0.0-5817-gbcea8c48 fa922c444906 89831375883a 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (2m) - 2m 12.6M 8559M 18.0.0-5817-gbcea8c48 fa922c444906 4728dbebae96 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi114 running (2m) - 2m 44.9M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 8aceaaa99f71 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi114 running (109s) - 108s 37.5M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 2ca7d4217927 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi114 running (93s) - 92s 15.0M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 55f470a2fedc 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi139 running (77s) - 77s 40.4M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 dab10c53ccc8 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi139 running (61s) - 61s 34.5M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 c1728e0f19e7 2023-10-03T07:13:12.032 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi139 running (45s) - 45s 11.6M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 7c7431b44a2d 2023-10-03T07:13:12.431 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-03T07:13:12.679 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:12.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:12 smithi114 ceph-mon[109024]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:12.808 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:12 smithi031 ceph-mon[107170]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:12 smithi139 ceph-mon[109290]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:13.428 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:13 smithi031 ceph-mon[107170]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:13.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:13 smithi114 ceph-mon[109024]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:13.835 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-03T07:13:13.835 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 1/1 - 2m smithi031=x;count:1 2023-10-03T07:13:13.835 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 - 3m smithi031:172.21.15.31=a;smithi114:172.21.15.114=b;smithi139:172.21.15.139=c;count:3 2023-10-03T07:13:13.835 INFO:teuthology.orchestra.run.smithi031.stdout:osd 8 - - 2023-10-03T07:13:13.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:13 smithi139 ceph-mon[109290]: from='client.14523 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:14.220 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch host ls' 2023-10-03T07:13:14.471 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:14.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:14 smithi031 ceph-mon[107170]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:14.683 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:14 smithi031 ceph-mon[107170]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:14 smithi114 ceph-mon[109024]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:14.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:14 smithi114 ceph-mon[109024]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:14 smithi139 ceph-mon[109290]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:14 smithi139 ceph-mon[109290]: from='client.14529 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:15.615 INFO:teuthology.orchestra.run.smithi031.stdout:HOST ADDR LABELS STATUS 2023-10-03T07:13:15.615 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 172.21.15.31 2023-10-03T07:13:15.616 INFO:teuthology.orchestra.run.smithi031.stdout:smithi114 172.21.15.114 2023-10-03T07:13:15.616 INFO:teuthology.orchestra.run.smithi031.stdout:smithi139 172.21.15.139 2023-10-03T07:13:15.616 INFO:teuthology.orchestra.run.smithi031.stdout:3 hosts in cluster 2023-10-03T07:13:15.995 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch device ls' 2023-10-03T07:13:16.244 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:16.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:16 smithi031 ceph-mon[107170]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:16.745 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:16 smithi031 ceph-mon[107170]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:16 smithi139 ceph-mon[109290]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:16 smithi139 ceph-mon[109290]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:16.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:16 smithi114 ceph-mon[109024]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:16.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:16 smithi114 ceph-mon[109024]: from='client.14535 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:17.437 INFO:teuthology.orchestra.run.smithi031.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2023-10-03T07:13:17.437 INFO:teuthology.orchestra.run.smithi031.stdout:smithi031 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT53310008400BGN 372G No 2m ago LVM detected, locked 2023-10-03T07:13:17.437 INFO:teuthology.orchestra.run.smithi031.stdout:smithi114 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WB400BGN 372G No 94s ago LVM detected, locked 2023-10-03T07:13:17.437 INFO:teuthology.orchestra.run.smithi031.stdout:smithi139 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500N6400BGN 372G No 46s ago LVM detected, locked 2023-10-03T07:13:17.819 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd lspools' 2023-10-03T07:13:18.066 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:18 smithi031 ceph-mon[107170]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:18.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:18 smithi031 ceph-mon[107170]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:18 smithi139 ceph-mon[109290]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:18 smithi139 ceph-mon[109290]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:18 smithi114 ceph-mon[109024]: from='client.14538 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:18 smithi114 ceph-mon[109024]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:19.227 INFO:teuthology.orchestra.run.smithi031.stdout:1 .mgr 2023-10-03T07:13:19.636 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool create mypool' 2023-10-03T07:13:19.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:19 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/2893193968' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:19.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:19 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/2893193968' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:19.883 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:19.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:19 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/2893193968' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:20.567 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:20 smithi031 ceph-mon[107170]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:20.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:20 smithi139 ceph-mon[109290]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:20.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:20 smithi114 ceph-mon[109024]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:21.571 INFO:teuthology.orchestra.run.smithi031.stderr:pool 'mypool' created 2023-10-03T07:13:21.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:21 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "mypool"}]: dispatch 2023-10-03T07:13:21.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:21 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "mypool"}]: dispatch 2023-10-03T07:13:21.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool application enable mypool rbd' 2023-10-03T07:13:21.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:21 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "mypool"}]: dispatch 2023-10-03T07:13:22.228 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:22.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:22 smithi031 ceph-mon[107170]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:22.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:22 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2023-10-03T07:13:22.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:22 smithi031 ceph-mon[107170]: osdmap e50: 8 total, 8 up, 8 in 2023-10-03T07:13:22.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:22 smithi139 ceph-mon[109290]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:22.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:22 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2023-10-03T07:13:22.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:22 smithi139 ceph-mon[109290]: osdmap e50: 8 total, 8 up, 8 in 2023-10-03T07:13:22.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:22 smithi114 ceph-mon[109024]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:22 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3923920101' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "mypool"}]': finished 2023-10-03T07:13:22.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:22 smithi114 ceph-mon[109024]: osdmap e50: 8 total, 8 up, 8 in 2023-10-03T07:13:23.579 INFO:teuthology.orchestra.run.smithi031.stderr:enabled application 'rbd' on pool 'mypool' 2023-10-03T07:13:23.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:23 smithi139 ceph-mon[109290]: osdmap e51: 8 total, 8 up, 8 in 2023-10-03T07:13:23.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:23 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]: dispatch 2023-10-03T07:13:23.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:23 smithi031 ceph-mon[107170]: osdmap e51: 8 total, 8 up, 8 in 2023-10-03T07:13:23.877 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:23 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]: dispatch 2023-10-03T07:13:23.977 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd lspools' 2023-10-03T07:13:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:23 smithi114 ceph-mon[109024]: osdmap e51: 8 total, 8 up, 8 in 2023-10-03T07:13:23.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:23 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]: dispatch 2023-10-03T07:13:24.225 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:24.672 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:24 smithi031 ceph-mon[107170]: pgmap v158: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:24.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:24 smithi031 ceph-mon[107170]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-03T07:13:24.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:24 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]': finished 2023-10-03T07:13:24.673 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:24 smithi031 ceph-mon[107170]: osdmap e52: 8 total, 8 up, 8 in 2023-10-03T07:13:24.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:24 smithi139 ceph-mon[109290]: pgmap v158: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:24.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:24 smithi139 ceph-mon[109290]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-03T07:13:24.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:24 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]': finished 2023-10-03T07:13:24.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:24 smithi139 ceph-mon[109290]: osdmap e52: 8 total, 8 up, 8 in 2023-10-03T07:13:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:24 smithi114 ceph-mon[109024]: pgmap v158: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:24 smithi114 ceph-mon[109024]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2023-10-03T07:13:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:24 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3000440651' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "mypool", "app": "rbd"}]': finished 2023-10-03T07:13:24.994 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:24 smithi114 ceph-mon[109024]: osdmap e52: 8 total, 8 up, 8 in 2023-10-03T07:13:25.424 INFO:teuthology.orchestra.run.smithi031.stdout:1 .mgr 2023-10-03T07:13:25.424 INFO:teuthology.orchestra.run.smithi031.stdout:2 mypool 2023-10-03T07:13:25.702 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:25 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/4073035524' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:25.758 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph osd pool ls detail' 2023-10-03T07:13:25.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:25 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/4073035524' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:25.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:25 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/4073035524' entity='client.admin' cmd=[{"prefix": "osd lspools"}]: dispatch 2023-10-03T07:13:26.007 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:26.727 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:26 smithi031 ceph-mon[107170]: pgmap v160: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:26.727 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:26 smithi031 ceph-mon[107170]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-10-03T07:13:26.727 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:26 smithi031 ceph-mon[107170]: Cluster is now healthy 2023-10-03T07:13:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:26 smithi139 ceph-mon[109290]: pgmap v160: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:26 smithi139 ceph-mon[109290]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-10-03T07:13:26.872 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:26 smithi139 ceph-mon[109290]: Cluster is now healthy 2023-10-03T07:13:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:26 smithi114 ceph-mon[109024]: pgmap v160: 33 pgs: 17 active+clean, 7 creating+activating, 6 creating+peering, 3 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:26 smithi114 ceph-mon[109024]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2023-10-03T07:13:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:26 smithi114 ceph-mon[109024]: Cluster is now healthy 2023-10-03T07:13:27.156 INFO:teuthology.orchestra.run.smithi031.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2023-10-03T07:13:27.157 INFO:teuthology.orchestra.run.smithi031.stdout:pool 2 'mypool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 52 flags hashpspool stripe_width 0 application rbd read_balance_score 1.75 2023-10-03T07:13:27.157 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:27.543 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph config set mgr mgr/cephadm/container_image_nvmeof quay.io/barakda1/nvmeof:0.0.3' 2023-10-03T07:13:27.787 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:27.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:27 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/4154782415' entity='client.admin' cmd=[{"prefix": "osd pool ls", "detail": "detail"}]: dispatch 2023-10-03T07:13:27.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:27 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/4154782415' entity='client.admin' cmd=[{"prefix": "osd pool ls", "detail": "detail"}]: dispatch 2023-10-03T07:13:27.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:27 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/4154782415' entity='client.admin' cmd=[{"prefix": "osd pool ls", "detail": "detail"}]: dispatch 2023-10-03T07:13:28.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:28 smithi031 ceph-mon[107170]: pgmap v161: 33 pgs: 26 active+clean, 7 creating+activating; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:28 smithi139 ceph-mon[109290]: pgmap v161: 33 pgs: 26 active+clean, 7 creating+activating; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:28 smithi114 ceph-mon[109024]: pgmap v161: 33 pgs: 26 active+clean, 7 creating+activating; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:29.271 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph config get mgr mgr/cephadm/container_image_nvmeof' 2023-10-03T07:13:29.528 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:29 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/1601207988' entity='client.admin' 2023-10-03T07:13:30.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:30.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:29 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:30.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:29 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/1601207988' entity='client.admin' 2023-10-03T07:13:30.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:30.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:30.250 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:30.251 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:29 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:30.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:29 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/1601207988' entity='client.admin' 2023-10-03T07:13:30.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:30.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:30.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:30.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:29 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:30.656 INFO:teuthology.orchestra.run.smithi031.stdout:quay.io/barakda1/nvmeof:0.0.3 2023-10-03T07:13:30.943 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:30 smithi031 ceph-mon[107170]: pgmap v162: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:30.943 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:30 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3302189965' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/container_image_nvmeof"}]: dispatch 2023-10-03T07:13:31.042 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch apply nvmeof mypool --placement="1 $(hostname)"' 2023-10-03T07:13:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:30 smithi114 ceph-mon[109024]: pgmap v162: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:31.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:30 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3302189965' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/container_image_nvmeof"}]: dispatch 2023-10-03T07:13:31.292 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:31.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:30 smithi139 ceph-mon[109290]: pgmap v162: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:31.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:30 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3302189965' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/cephadm/container_image_nvmeof"}]: dispatch 2023-10-03T07:13:32.518 INFO:teuthology.orchestra.run.smithi031.stdout:Scheduled nvmeof.mypool update... 2023-10-03T07:13:32.925 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ps --refresh' 2023-10-03T07:13:32.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: pgmap v163: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:32.956 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]: dispatch 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]': finished 2023-10-03T07:13:32.957 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:32 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:33.182 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: pgmap v163: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:33.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]: dispatch 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]': finished 2023-10-03T07:13:33.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:32 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:33.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: pgmap v163: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:33.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]: dispatch 2023-10-03T07:13:33.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "client.nvmeof.mypool.smithi031.ddmnhv", "caps": ["mon", "profile rbd", "osd", "allow all tag rbd *=*"]}]': finished 2023-10-03T07:13:33.373 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:32 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:33.940 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:33 smithi031 ceph-mon[107170]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "placement": "1 smithi031", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:33.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:33 smithi031 ceph-mon[107170]: Saving service nvmeof.mypool spec with placement smithi031;count:1 2023-10-03T07:13:33.941 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:33 smithi031 ceph-mon[107170]: Deploying daemon nvmeof.mypool.smithi031.ddmnhv on smithi031 2023-10-03T07:13:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:33 smithi114 ceph-mon[109024]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "placement": "1 smithi031", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:33 smithi114 ceph-mon[109024]: Saving service nvmeof.mypool spec with placement smithi031;count:1 2023-10-03T07:13:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:33 smithi114 ceph-mon[109024]: Deploying daemon nvmeof.mypool.smithi031.ddmnhv on smithi031 2023-10-03T07:13:34.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:33 smithi139 ceph-mon[109290]: from='client.14586 -' entity='client.admin' cmd=[{"prefix": "orch apply nvmeof", "pool": "mypool", "placement": "1 smithi031", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:34.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:33 smithi139 ceph-mon[109290]: Saving service nvmeof.mypool spec with placement smithi031;count:1 2023-10-03T07:13:34.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:33 smithi139 ceph-mon[109290]: Deploying daemon nvmeof.mypool.smithi031.ddmnhv on smithi031 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:mgr.x smithi031 *:9283,8765 running (4m) - 4m 489M - 18.0.0-5817-gbcea8c48 fa922c444906 2e68f5c77307 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:mon.a smithi031 running (4m) - 4m 47.6M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 1b0e77539eb0 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:mon.b smithi114 running (3m) - 3m 43.4M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 97e2dcb1ebad 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:mon.c smithi139 running (3m) - 3m 47.2M 2048M 18.0.0-5817-gbcea8c48 fa922c444906 4fb98de11a35 2023-10-03T07:13:34.452 INFO:teuthology.orchestra.run.smithi031.stdout:osd.0 smithi031 running (3m) - 2m 29.0M 8559M 18.0.0-5817-gbcea8c48 fa922c444906 89831375883a 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.1 smithi031 running (2m) - 2m 12.6M 8559M 18.0.0-5817-gbcea8c48 fa922c444906 4728dbebae96 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.2 smithi114 running (2m) - 2m 44.9M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 8aceaaa99f71 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.3 smithi114 running (2m) - 2m 37.5M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 2ca7d4217927 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.4 smithi114 running (115s) - 115s 15.0M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 55f470a2fedc 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.5 smithi139 running (99s) - 99s 40.4M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 dab10c53ccc8 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.6 smithi139 running (83s) - 83s 34.5M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 c1728e0f19e7 2023-10-03T07:13:34.453 INFO:teuthology.orchestra.run.smithi031.stdout:osd.7 smithi139 running (68s) - 68s 11.6M 7069M 18.0.0-5817-gbcea8c48 fa922c444906 7c7431b44a2d 2023-10-03T07:13:34.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph versions' 2023-10-03T07:13:35.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:34 smithi031 ceph-mon[107170]: pgmap v164: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:35.092 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:35.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:34 smithi114 ceph-mon[109024]: pgmap v164: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:35.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:34 smithi139 ceph-mon[109290]: pgmap v164: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:36.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:35 smithi114 ceph-mon[109024]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:36.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:35 smithi031 ceph-mon[107170]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:36.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:35 smithi139 ceph-mon[109290]: from='client.14592 -' entity='client.admin' cmd=[{"prefix": "orch ps", "refresh": true, "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:36.570 INFO:teuthology.orchestra.run.smithi031.stdout:{ 2023-10-03T07:13:36.570 INFO:teuthology.orchestra.run.smithi031.stdout: "mon": { 2023-10-03T07:13:36.570 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 3 2023-10-03T07:13:36.570 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "mgr": { 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 1 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "osd": { 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 8 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: }, 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "overall": { 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: "ceph version 18.0.0-5817-gbcea8c48 (bcea8c48517ff154f392584022efa0718f51c9f2) reef (dev)": 12 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout: } 2023-10-03T07:13:36.571 INFO:teuthology.orchestra.run.smithi031.stdout:} 2023-10-03T07:13:36.959 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph -s' 2023-10-03T07:13:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:36 smithi114 ceph-mon[109024]: pgmap v165: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:37.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:36 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/997866525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-10-03T07:13:37.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:36 smithi031 ceph-mon[107170]: pgmap v165: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:37.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:36 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/997866525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-10-03T07:13:37.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:36 smithi139 ceph-mon[109290]: pgmap v165: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:37.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:36 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/997866525' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2023-10-03T07:13:38.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:37 smithi114 ceph-mon[109024]: pgmap v166: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:38.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:37 smithi031 ceph-mon[107170]: pgmap v166: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:38.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:37 smithi139 ceph-mon[109290]: pgmap v166: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:40.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:40 smithi114 ceph-mon[109024]: pgmap v167: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:40.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:40 smithi031 ceph-mon[107170]: pgmap v167: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:40.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:40 smithi139 ceph-mon[109290]: pgmap v167: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:41.827 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:42.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:42 smithi031 ceph-mon[107170]: pgmap v168: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:42.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:42 smithi139 ceph-mon[109290]: pgmap v168: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:42.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:42 smithi114 ceph-mon[109024]: pgmap v168: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:43.834 INFO:teuthology.orchestra.run.smithi031.stdout: cluster: 2023-10-03T07:13:43.834 INFO:teuthology.orchestra.run.smithi031.stdout: id: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: health: HEALTH_OK 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: services: 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: mon: 3 daemons, quorum a,c,b (age 3m) 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: mgr: x(active, since 4m) 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: osd: 8 osds: 8 up (since 71s), 8 in (since 85s) 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: data: 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: pools: 2 pools, 33 pgs 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: objects: 2 objects, 577 KiB 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: usage: 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:43.835 INFO:teuthology.orchestra.run.smithi031.stdout: pgs: 33 active+clean 2023-10-03T07:13:43.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:43.836 INFO:teuthology.orchestra.run.smithi031.stdout: progress: 2023-10-03T07:13:43.836 INFO:teuthology.orchestra.run.smithi031.stdout: Updating nvmeof.mypool deployment (+1 -> 1) (0s) 2023-10-03T07:13:43.836 INFO:teuthology.orchestra.run.smithi031.stdout: [............................] 2023-10-03T07:13:43.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:44.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: Checking pool "mypool" exists for service nvmeof.mypool 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: pgmap v169: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='client.? 172.21.15.31:0/3163164911' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:44 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: Checking pool "mypool" exists for service nvmeof.mypool 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:44.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: pgmap v169: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:44.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='client.? 172.21.15.31:0/3163164911' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-03T07:13:44.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.318 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:44 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.469 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'ceph orch ls' 2023-10-03T07:13:44.621 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: Checking pool "mypool" exists for service nvmeof.mypool 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: pgmap v169: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:44.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='client.? 172.21.15.31:0/3163164911' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-10-03T07:13:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:44 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:44.760 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:45.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.505 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.506 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:45.506 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:45.506 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.506 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:45 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:45.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.622 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:45.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:45.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.623 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:45 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:45.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:45.744 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:45.745 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:45 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-10-03T07:13:45.963 INFO:teuthology.orchestra.run.smithi031.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2023-10-03T07:13:45.963 INFO:teuthology.orchestra.run.smithi031.stdout:mgr 1/1 - 3m smithi031=x;count:1 2023-10-03T07:13:45.963 INFO:teuthology.orchestra.run.smithi031.stdout:mon 3/3 - 3m smithi031:172.21.15.31=a;smithi114:172.21.15.114=b;smithi139:172.21.15.139=c;count:3 2023-10-03T07:13:45.963 INFO:teuthology.orchestra.run.smithi031.stdout:nvmeof.mypool ?:4420,5500,8009 1/1 - 13s smithi031;count:1 2023-10-03T07:13:45.964 INFO:teuthology.orchestra.run.smithi031.stdout:osd 8 - - 2023-10-03T07:13:46.784 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'rbd create mypool/myimage --size 8Gi' 2023-10-03T07:13:46.814 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:46 smithi031 ceph-mon[107170]: pgmap v170: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:46.815 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:46 smithi031 ceph-mon[107170]: mgrmap e15: x(active, since 4m) 2023-10-03T07:13:47.037 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:47.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:46 smithi139 ceph-mon[109290]: pgmap v170: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:47.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:46 smithi139 ceph-mon[109290]: mgrmap e15: x(active, since 4m) 2023-10-03T07:13:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:46 smithi114 ceph-mon[109024]: pgmap v170: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:47.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:46 smithi114 ceph-mon[109024]: mgrmap e15: x(active, since 4m) 2023-10-03T07:13:48.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:48.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:48.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:48.068 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:47 smithi031 ceph-mon[107170]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:48.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:48.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:48.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:47 smithi139 ceph-mon[109290]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='client.14622 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2023-10-03T07:13:48.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-10-03T07:13:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2023-10-03T07:13:48.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:47 smithi114 ceph-mon[109024]: from='mgr.14152 172.21.15.31:0/2433509987' entity='mgr.x' 2023-10-03T07:13:49.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:48 smithi139 ceph-mon[109290]: pgmap v171: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:49.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:48 smithi114 ceph-mon[109024]: pgmap v171: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:49.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:48 smithi031 ceph-mon[107170]: pgmap v171: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2023-10-03T07:13:50.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:49 smithi139 ceph-mon[109290]: osdmap e53: 8 total, 8 up, 8 in 2023-10-03T07:13:50.138 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:49 smithi031 ceph-mon[107170]: osdmap e53: 8 total, 8 up, 8 in 2023-10-03T07:13:50.242 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'rbd ls mypool' 2023-10-03T07:13:50.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:49 smithi114 ceph-mon[109024]: osdmap e53: 8 total, 8 up, 8 in 2023-10-03T07:13:50.535 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:50.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:50 smithi031 ceph-mon[107170]: pgmap v173: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2023-10-03T07:13:50.979 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:50 smithi031 ceph-mon[107170]: osdmap e54: 8 total, 8 up, 8 in 2023-10-03T07:13:51.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:50 smithi139 ceph-mon[109290]: pgmap v173: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2023-10-03T07:13:51.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:50 smithi139 ceph-mon[109290]: osdmap e54: 8 total, 8 up, 8 in 2023-10-03T07:13:51.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:50 smithi114 ceph-mon[109024]: pgmap v173: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 102 B/s rd, 102 B/s wr, 0 op/s 2023-10-03T07:13:51.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:50 smithi114 ceph-mon[109024]: osdmap e54: 8 total, 8 up, 8 in 2023-10-03T07:13:51.347 INFO:teuthology.orchestra.run.smithi031.stdout:myimage 2023-10-03T07:13:51.701 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:bcea8c48517ff154f392584022efa0718f51c9f2 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 -- bash -c 'set -ex 2023-10-03T07:13:51.701 DEBUG:teuthology.orchestra.run.smithi031:> # 2023-10-03T07:13:51.701 DEBUG:teuthology.orchestra.run.smithi031:> cat /proc/cpuinfo 2023-10-03T07:13:51.701 DEBUG:teuthology.orchestra.run.smithi031:> ' 2023-10-03T07:13:51.952 INFO:teuthology.orchestra.run.smithi031.stderr:Inferring config /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/mon.a/config 2023-10-03T07:13:52.808 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 0 2023-10-03T07:13:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.809 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.049 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 0 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.810 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 0 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 0 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.811 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 1 2023-10-03T07:13:52.812 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.000 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.813 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 1 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 2 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 2 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.814 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.815 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 2 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.816 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.000 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.817 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 2 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 4 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 4 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.818 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.819 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 3 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.000 2023-10-03T07:13:52.820 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 3 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 6 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 6 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.821 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.822 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 4 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.823 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.000 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 0 2023-10-03T07:13:52.824 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 1 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 1 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.825 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 5 2023-10-03T07:13:52.826 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.000 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.827 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 1 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 3 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 3 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.828 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 6 2023-10-03T07:13:52.829 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.078 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.830 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 2 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 5 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 5 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.831 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.832 INFO:teuthology.orchestra.run.smithi031.stdout:processor : 7 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:vendor_id : GenuineIntel 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:cpu family : 6 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:model : 63 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:model name : Intel(R) Xeon(R) CPU E5-1620 v3 @ 3.50GHz 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:stepping : 2 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:microcode : 0x49 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:cpu MHz : 3500.134 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:cache size : 10240 KB 2023-10-03T07:13:52.833 INFO:teuthology.orchestra.run.smithi031.stdout:physical id : 0 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:siblings : 8 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:core id : 3 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:cpu cores : 4 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:apicid : 7 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:initial apicid : 7 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:fpu : yes 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:fpu_exception : yes 2023-10-03T07:13:52.834 INFO:teuthology.orchestra.run.smithi031.stdout:cpuid level : 15 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:wp : yes 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit mmio_stale_data 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:bogomips : 7000.09 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:clflush size : 64 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:cache_alignment : 64 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:address sizes : 46 bits physical, 48 bits virtual 2023-10-03T07:13:52.835 INFO:teuthology.orchestra.run.smithi031.stdout:power management: 2023-10-03T07:13:52.836 INFO:teuthology.orchestra.run.smithi031.stdout: 2023-10-03T07:13:52.836 INFO:teuthology.orchestra.run.smithi031.stderr:+ cat /proc/cpuinfo 2023-10-03T07:13:53.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:52 smithi139 ceph-mon[109290]: pgmap v175: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2023-10-03T07:13:53.157 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:52 smithi031 ceph-mon[107170]: pgmap v175: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2023-10-03T07:13:53.159 INFO:teuthology.run_tasks:Running task workunit... 2023-10-03T07:13:53.171 INFO:tasks.workunit:Pulling workunits from ref b6b4a26de3cb4337d4479f11f356af936649f829 2023-10-03T07:13:53.171 INFO:tasks.workunit:Making a separate scratch dir for every client... 2023-10-03T07:13:53.172 DEBUG:teuthology.orchestra.run.smithi031:> stat -- /home/ubuntu/cephtest/mnt.0 2023-10-03T07:13:53.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:13:53.192 INFO:teuthology.orchestra.run.smithi031.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2023-10-03T07:13:53.193 DEBUG:teuthology.orchestra.run.smithi031:> mkdir -- /home/ubuntu/cephtest/mnt.0 2023-10-03T07:13:53.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:52 smithi114 ceph-mon[109024]: pgmap v175: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 127 B/s rd, 127 B/s wr, 0 op/s 2023-10-03T07:13:53.252 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2023-10-03T07:13:53.252 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2023-10-03T07:13:53.311 INFO:tasks.workunit:timeout=3h 2023-10-03T07:13:53.311 INFO:tasks.workunit:cleanup=True 2023-10-03T07:13:53.311 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/barakda/ceph /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout b6b4a26de3cb4337d4479f11f356af936649f829 2023-10-03T07:13:53.370 INFO:tasks.workunit.client.0.smithi031.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2023-10-03T07:13:55.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:54 smithi139 ceph-mon[109290]: pgmap v176: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 895 B/s wr, 2 op/s 2023-10-03T07:13:55.122 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:54 smithi139 ceph-mon[109290]: osdmap e55: 8 total, 8 up, 8 in 2023-10-03T07:13:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:54 smithi114 ceph-mon[109024]: pgmap v176: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 895 B/s wr, 2 op/s 2023-10-03T07:13:55.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:54 smithi114 ceph-mon[109024]: osdmap e55: 8 total, 8 up, 8 in 2023-10-03T07:13:55.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:54 smithi031 ceph-mon[107170]: pgmap v176: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 767 B/s rd, 895 B/s wr, 2 op/s 2023-10-03T07:13:55.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:54 smithi031 ceph-mon[107170]: osdmap e55: 8 total, 8 up, 8 in 2023-10-03T07:13:57.121 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:56 smithi139 ceph-mon[109290]: pgmap v178: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 923 B/s wr, 2 op/s 2023-10-03T07:13:57.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:56 smithi114 ceph-mon[109024]: pgmap v178: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 923 B/s wr, 2 op/s 2023-10-03T07:13:57.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:56 smithi031 ceph-mon[107170]: pgmap v178: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 769 B/s rd, 923 B/s wr, 2 op/s 2023-10-03T07:13:59.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:13:58 smithi114 ceph-mon[109024]: pgmap v179: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2023-10-03T07:13:59.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:13:58 smithi031 ceph-mon[107170]: pgmap v179: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2023-10-03T07:13:59.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:13:58 smithi139 ceph-mon[109290]: pgmap v179: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 767 B/s wr, 1 op/s 2023-10-03T07:14:00.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:00 smithi114 ceph-mon[109024]: pgmap v180: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 636 B/s wr, 1 op/s 2023-10-03T07:14:00.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:00 smithi031 ceph-mon[107170]: pgmap v180: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 636 B/s wr, 1 op/s 2023-10-03T07:14:00.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:00 smithi139 ceph-mon[109290]: pgmap v180: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 636 B/s rd, 636 B/s wr, 1 op/s 2023-10-03T07:14:02.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:02 smithi031 ceph-mon[107170]: pgmap v181: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 614 B/s wr, 1 op/s 2023-10-03T07:14:02.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:02 smithi139 ceph-mon[109290]: pgmap v181: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 614 B/s wr, 1 op/s 2023-10-03T07:14:02.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:02 smithi114 ceph-mon[109024]: pgmap v181: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 614 B/s wr, 1 op/s 2023-10-03T07:14:04.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:04 smithi031 ceph-mon[107170]: pgmap v182: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-10-03T07:14:04.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:04 smithi139 ceph-mon[109290]: pgmap v182: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-10-03T07:14:04.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:04 smithi114 ceph-mon[109024]: pgmap v182: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 204 B/s rd, 0 op/s 2023-10-03T07:14:06.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:06 smithi031 ceph-mon[107170]: pgmap v183: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 184 B/s rd, 0 op/s 2023-10-03T07:14:06.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:06 smithi139 ceph-mon[109290]: pgmap v183: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 184 B/s rd, 0 op/s 2023-10-03T07:14:06.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:06 smithi114 ceph-mon[109024]: pgmap v183: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 184 B/s rd, 0 op/s 2023-10-03T07:14:08.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:08 smithi139 ceph-mon[109290]: pgmap v184: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:08.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:08 smithi114 ceph-mon[109024]: pgmap v184: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:09.067 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:08 smithi031 ceph-mon[107170]: pgmap v184: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:10.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:10 smithi114 ceph-mon[109024]: pgmap v185: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:10.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:10 smithi031 ceph-mon[107170]: pgmap v185: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:10.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:10 smithi139 ceph-mon[109290]: pgmap v185: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:12.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:12 smithi031 ceph-mon[107170]: pgmap v186: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:12.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:12 smithi139 ceph-mon[109290]: pgmap v186: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:12.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:12 smithi114 ceph-mon[109024]: pgmap v186: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:14.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:14 smithi031 ceph-mon[107170]: pgmap v187: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:14.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:14 smithi139 ceph-mon[109290]: pgmap v187: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:14.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:14 smithi114 ceph-mon[109024]: pgmap v187: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:16.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:16 smithi031 ceph-mon[107170]: pgmap v188: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:16.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:16 smithi139 ceph-mon[109290]: pgmap v188: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:16.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:16 smithi114 ceph-mon[109024]: pgmap v188: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:18.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:18 smithi139 ceph-mon[109290]: pgmap v189: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:18.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:18 smithi114 ceph-mon[109024]: pgmap v189: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:19.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:18 smithi031 ceph-mon[107170]: pgmap v189: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:20.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:20 smithi114 ceph-mon[109024]: pgmap v190: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:20.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:20 smithi031 ceph-mon[107170]: pgmap v190: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:20.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:20 smithi139 ceph-mon[109290]: pgmap v190: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:22.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:22 smithi031 ceph-mon[107170]: pgmap v191: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:22.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:22 smithi139 ceph-mon[109290]: pgmap v191: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:22.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:22 smithi114 ceph-mon[109024]: pgmap v191: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:24.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:24 smithi031 ceph-mon[107170]: pgmap v192: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:24.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:24 smithi139 ceph-mon[109290]: pgmap v192: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:24.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:24 smithi114 ceph-mon[109024]: pgmap v192: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:26.817 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:26 smithi031 ceph-mon[107170]: pgmap v193: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:26.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:26 smithi139 ceph-mon[109290]: pgmap v193: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:26.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:26 smithi114 ceph-mon[109024]: pgmap v193: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:28.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:28 smithi139 ceph-mon[109290]: pgmap v194: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:28.993 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:28 smithi114 ceph-mon[109024]: pgmap v194: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:29.066 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:28 smithi031 ceph-mon[107170]: pgmap v194: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:30.743 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:30 smithi114 ceph-mon[109024]: pgmap v195: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:30.789 INFO:tasks.workunit.client.0.smithi031.stderr:Updating files: 97% (11706/12039) Updating files: 98% (11799/12039) Updating files: 99% (11919/12039) Updating files: 100% (12039/12039) Updating files: 100% (12039/12039), done. 2023-10-03T07:14:30.819 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:30 smithi031 ceph-mon[107170]: pgmap v195: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:30.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:30 smithi139 ceph-mon[109290]: pgmap v195: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 0 op/s 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr:Note: switching to 'b6b4a26de3cb4337d4479f11f356af936649f829'. 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr:changes and commit them, and you can discard any commits you make in this 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr:state without impacting any branches by switching back to a branch. 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.388 INFO:tasks.workunit.client.0.smithi031.stderr:If you want to create a new branch to retain commits you create, you may 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr:do so (now or later) by using -c with the switch command. Example: 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr: git switch -c 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr:Or undo this operation with: 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.389 INFO:tasks.workunit.client.0.smithi031.stderr: git switch - 2023-10-03T07:14:31.390 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.390 INFO:tasks.workunit.client.0.smithi031.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2023-10-03T07:14:31.390 INFO:tasks.workunit.client.0.smithi031.stderr: 2023-10-03T07:14:31.390 INFO:tasks.workunit.client.0.smithi031.stderr:HEAD is now at b6b4a26de3 update install script 2023-10-03T07:14:31.391 DEBUG:teuthology.orchestra.run.smithi031:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2023-10-03T07:14:31.459 INFO:tasks.workunit.client.0.smithi031.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2023-10-03T07:14:31.461 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T07:14:31.461 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2023-10-03T07:14:32.004 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2023-10-03T07:14:32.065 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2023-10-03T07:14:32.106 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2023-10-03T07:14:32.108 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T07:14:32.108 INFO:tasks.workunit.client.0.smithi031.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2023-10-03T07:14:32.149 INFO:tasks.workunit.client.0.smithi031.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2023-10-03T07:14:32.152 DEBUG:teuthology.orchestra.run.smithi031:> set -ex 2023-10-03T07:14:32.153 DEBUG:teuthology.orchestra.run.smithi031:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2023-10-03T07:14:32.215 INFO:tasks.workunit:Running workunits matching cephadm/test_nvme_etc_hosts.sh on client.0... 2023-10-03T07:14:32.216 INFO:tasks.workunit:Running workunit cephadm/test_nvme_etc_hosts.sh... 2023-10-03T07:14:32.217 DEBUG:teuthology.orchestra.run.smithi031:workunit test cephadm/test_nvme_etc_hosts.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6b4a26de3cb4337d4479f11f356af936649f829 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_nvme_etc_hosts.sh 2023-10-03T07:14:32.275 DEBUG:teuthology.orchestra.run:got remote process result: 127 2023-10-03T07:14:32.276 INFO:tasks.workunit.client.0.smithi031.stderr:bash: adjust-ulimits: command not found 2023-10-03T07:14:32.276 INFO:tasks.workunit:Stopping ['cephadm/test_nvme_etc_hosts.sh'] on client.0... 2023-10-03T07:14:32.276 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2023-10-03T07:14:32.600 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:32 smithi031 ceph-mon[107170]: pgmap v196: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:32.602 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephadm/test_nvme_etc_hosts.sh) on smithi031 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6b4a26de3cb4337d4479f11f356af936649f829 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_nvme_etc_hosts.sh' 2023-10-03T07:14:32.737 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=4637935d25af414d8cc9d6d3c20414d9 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa/tasks/workunit.py", line 129, in task p.spawn(_run_tests, ctx, refspec, role, tests, File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/github.com_barakda_ceph_b6b4a26de3cb4337d4479f11f356af936649f829/qa/tasks/workunit.py", line 424, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test cephadm/test_nvme_etc_hosts.sh) on smithi031 with status 127: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6b4a26de3cb4337d4479f11f356af936649f829 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_nvme_etc_hosts.sh' 2023-10-03T07:14:32.739 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-10-03T07:14:32.749 INFO:tasks.cephadm:Teardown begin 2023-10-03T07:14:32.749 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:14:32.802 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:14:32.838 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:32 smithi114 ceph-mon[109024]: pgmap v196: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:32.842 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:14:32.871 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:32 smithi139 ceph-mon[109290]: pgmap v196: 33 pgs: 33 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 170 B/s rd, 0 op/s 2023-10-03T07:14:32.881 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-10-03T07:14:32.881 DEBUG:teuthology.orchestra.run.smithi031:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-03T07:14:32.901 DEBUG:teuthology.orchestra.run.smithi114:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-03T07:14:32.922 DEBUG:teuthology.orchestra.run.smithi139:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-10-03T07:14:32.944 INFO:tasks.cephadm:Stopping all daemons... 2023-10-03T07:14:32.944 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2023-10-03T07:14:32.944 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a 2023-10-03T07:14:33.316 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:32 smithi031 systemd[1]: Stopping Ceph mon.a for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:33.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:33 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a[107166]: 2023-10-03T07:14:33.169+0000 7f2887390700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:33.317 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:33 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a[107166]: 2023-10-03T07:14:33.169+0000 7f2887390700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2023-10-03T07:14:33.816 INFO:journalctl@ceph.mon.a.smithi031.stdout:Oct 03 07:14:33 smithi031 bash[134492]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-a 2023-10-03T07:14:33.869 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.a.service' 2023-10-03T07:14:33.920 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:33.921 INFO:tasks.cephadm.mon.a:Stopped mon.a 2023-10-03T07:14:33.921 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2023-10-03T07:14:33.921 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.b 2023-10-03T07:14:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:33 smithi114 systemd[1]: Stopping Ceph mon.b for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:34.243 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:34 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-b[109019]: 2023-10-03T07:14:34.129+0000 7f265720c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:34.244 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:34 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-b[109019]: 2023-10-03T07:14:34.129+0000 7f265720c700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2023-10-03T07:14:34.692 INFO:journalctl@ceph.mon.b.smithi114.stdout:Oct 03 07:14:34 smithi114 bash[124077]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-b 2023-10-03T07:14:34.699 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.b.service' 2023-10-03T07:14:34.773 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:34.773 INFO:tasks.cephadm.mon.c:Stopped mon.b 2023-10-03T07:14:34.773 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2023-10-03T07:14:34.773 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.c 2023-10-03T07:14:35.090 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:34 smithi139 systemd[1]: Stopping Ceph mon.c for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:35.371 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:35 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-c[109286]: 2023-10-03T07:14:35.089+0000 7f71d79f2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:35.372 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:35 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-c[109286]: 2023-10-03T07:14:35.089+0000 7f71d79f2700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2023-10-03T07:14:35.732 INFO:journalctl@ceph.mon.c.smithi139.stdout:Oct 03 07:14:35 smithi139 bash[125105]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mon-c 2023-10-03T07:14:35.738 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mon.c.service' 2023-10-03T07:14:35.795 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:35.795 INFO:tasks.cephadm.mon.c:Stopped mon.c 2023-10-03T07:14:35.795 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2023-10-03T07:14:35.795 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x 2023-10-03T07:14:36.152 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:14:35 smithi031 systemd[1]: Stopping Ceph mgr.x for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:36.152 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:14:36 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x[107393]: 2023-10-03T07:14:36.029+0000 7f3224b0e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:36.560 INFO:journalctl@ceph.mgr.x.smithi031.stdout:Oct 03 07:14:36 smithi031 bash[134729]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-mgr-x 2023-10-03T07:14:36.566 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@mgr.x.service' 2023-10-03T07:14:36.607 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:36.607 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2023-10-03T07:14:36.607 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2023-10-03T07:14:36.608 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.0 2023-10-03T07:14:36.816 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:14:36 smithi031 systemd[1]: Stopping Ceph osd.0 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:37.316 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:14:36 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:14:36.867+0000 7f3257376700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:37.317 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:14:36 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:14:36.867+0000 7f3257376700 -1 osd.0 55 *** Got signal Terminated *** 2023-10-03T07:14:37.317 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:14:36 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0[118492]: 2023-10-03T07:14:36.867+0000 7f3257376700 -1 osd.0 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:14:42.817 INFO:journalctl@ceph.osd.0.smithi031.stdout:Oct 03 07:14:42 smithi031 bash[134861]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-0 2023-10-03T07:14:43.645 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.0.service' 2023-10-03T07:14:43.691 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:43.691 INFO:tasks.cephadm.osd.0:Stopped osd.0 2023-10-03T07:14:43.691 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2023-10-03T07:14:43.691 DEBUG:teuthology.orchestra.run.smithi031:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.1 2023-10-03T07:14:43.931 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:14:43 smithi031 systemd[1]: Stopping Ceph osd.1 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:44.316 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:14:43 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:14:43.930+0000 7f10caf80700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:44.316 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:14:43 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:14:43.930+0000 7f10caf80700 -1 osd.1 55 *** Got signal Terminated *** 2023-10-03T07:14:44.317 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:14:43 smithi031 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1[122728]: 2023-10-03T07:14:43.930+0000 7f10caf80700 -1 osd.1 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:14:49.816 INFO:journalctl@ceph.osd.1.smithi031.stdout:Oct 03 07:14:49 smithi031 bash[135087]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-1 2023-10-03T07:14:50.659 DEBUG:teuthology.orchestra.run.smithi031:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.1.service' 2023-10-03T07:14:50.705 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:50.705 INFO:tasks.cephadm.osd.1:Stopped osd.1 2023-10-03T07:14:50.706 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2023-10-03T07:14:50.706 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.2 2023-10-03T07:14:50.993 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:14:50 smithi114 systemd[1]: Stopping Ceph osd.2 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:50.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:14:50 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:14:50.936+0000 7f790bdea700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:50.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:14:50 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:14:50.936+0000 7f790bdea700 -1 osd.2 55 *** Got signal Terminated *** 2023-10-03T07:14:50.994 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:14:50 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2[113029]: 2023-10-03T07:14:50.936+0000 7f790bdea700 -1 osd.2 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:14:56.743 INFO:journalctl@ceph.osd.2.smithi114.stdout:Oct 03 07:14:56 smithi114 bash[124229]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-2 2023-10-03T07:14:57.588 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.2.service' 2023-10-03T07:14:57.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:14:57.627 INFO:tasks.cephadm.osd.2:Stopped osd.2 2023-10-03T07:14:57.628 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2023-10-03T07:14:57.628 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.3 2023-10-03T07:14:57.866 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:14:57 smithi114 systemd[1]: Stopping Ceph osd.3 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:14:58.243 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:14:57 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:14:57.866+0000 7efd53baa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:14:58.243 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:14:57 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:14:57.866+0000 7efd53baa700 -1 osd.3 55 *** Got signal Terminated *** 2023-10-03T07:14:58.244 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:14:57 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3[117057]: 2023-10-03T07:14:57.866+0000 7efd53baa700 -1 osd.3 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:15:02.872 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:02 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:02.473+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:03.743 INFO:journalctl@ceph.osd.3.smithi114.stdout:Oct 03 07:15:03 smithi114 bash[124454]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-3 2023-10-03T07:15:03.871 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:03 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:03.477+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:03.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:03 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:03.471+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:04.621 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:04 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:04.514+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:04.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:04 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:04.346+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:04.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:04 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:04.423+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:04.629 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.3.service' 2023-10-03T07:15:04.669 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:15:04.669 INFO:tasks.cephadm.osd.3:Stopped osd.3 2023-10-03T07:15:04.669 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2023-10-03T07:15:04.670 DEBUG:teuthology.orchestra.run.smithi114:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.4 2023-10-03T07:15:04.912 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:04 smithi114 systemd[1]: Stopping Ceph osd.4 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:15:05.243 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:04 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:04.911+0000 7fc0b00f5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:15:05.244 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:04 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:04.911+0000 7fc0b00f5700 -1 osd.4 55 *** Got signal Terminated *** 2023-10-03T07:15:05.244 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:04 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:04.911+0000 7fc0b00f5700 -1 osd.4 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:15:05.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:05 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:05.359+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:05.622 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:05 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:05.486+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:05.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:05 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:05.435+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:06.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:06 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:06.319+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:06.622 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:06 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:06.507+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:06.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:06 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:06.475+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:06.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:06 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:06.420+0000 7fc0a9ee6700 -1 osd.4 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:40.192515+0000 front 2023-10-03T07:14:40.192574+0000 (oldest deadline 2023-10-03T07:15:05.492665+0000) 2023-10-03T07:15:07.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:07 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:07.326+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:07.622 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:07 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:07.541+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:07.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:07 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:07.493+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:07.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:07 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:07.395+0000 7fc0a9ee6700 -1 osd.4 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:40.192515+0000 front 2023-10-03T07:14:40.192574+0000 (oldest deadline 2023-10-03T07:15:05.492665+0000) 2023-10-03T07:15:08.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:08 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:08.312+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:08.622 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:08 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:08.544+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:08.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:08 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:08.502+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:08.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:08 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:08.349+0000 7fc0a9ee6700 -1 osd.4 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:40.192515+0000 front 2023-10-03T07:14:40.192574+0000 (oldest deadline 2023-10-03T07:15:05.492665+0000) 2023-10-03T07:15:09.532 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:09 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:09.281+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:09.743 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:09 smithi114 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4[121159]: 2023-10-03T07:15:09.358+0000 7fc0a9ee6700 -1 osd.4 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:40.192515+0000 front 2023-10-03T07:14:40.192574+0000 (oldest deadline 2023-10-03T07:15:05.492665+0000) 2023-10-03T07:15:09.871 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:09 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:09.536+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:09.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:09 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:09.532+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:10.538 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:10 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:10.279+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:10.538 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:10 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:10.279+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:10.871 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:10 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:10.556+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:10.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:10 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:10.538+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:10.993 INFO:journalctl@ceph.osd.4.smithi114.stdout:Oct 03 07:15:10 smithi114 bash[124679]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-4 2023-10-03T07:15:11.621 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:11 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:11.507+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:11.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:11 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:11.298+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:11.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:11 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:11.298+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:11.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:11 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:11.516+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:11.770 DEBUG:teuthology.orchestra.run.smithi114:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.4.service' 2023-10-03T07:15:11.809 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:15:11.809 INFO:tasks.cephadm.osd.4:Stopped osd.4 2023-10-03T07:15:11.809 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2023-10-03T07:15:11.809 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.5 2023-10-03T07:15:12.121 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:11 smithi139 systemd[1]: Stopping Ceph osd.5 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:15:12.121 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:12.037+0000 7f40ccf1f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:15:12.121 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:12.037+0000 7f40ccf1f700 -1 osd.5 55 *** Got signal Terminated *** 2023-10-03T07:15:12.122 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:12.037+0000 7f40ccf1f700 -1 osd.5 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:15:12.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:12.305+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:12.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:12.305+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:12.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:12.524+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:12.623 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:12.478+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:12.623 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:12 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:12.478+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:47.154433+0000 front 2023-10-03T07:14:47.154391+0000 (oldest deadline 2023-10-03T07:15:11.854478+0000) 2023-10-03T07:15:13.621 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:13 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:13.501+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:13.622 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:13 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:13.501+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:47.154433+0000 front 2023-10-03T07:14:47.154391+0000 (oldest deadline 2023-10-03T07:15:11.854478+0000) 2023-10-03T07:15:13.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:13 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:13.350+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:13.623 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:13 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:13.350+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:13.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:13 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:13.534+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:14.621 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:14.468+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:14.621 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:14.468+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:47.154433+0000 front 2023-10-03T07:14:47.154391+0000 (oldest deadline 2023-10-03T07:15:11.854478+0000) 2023-10-03T07:15:14.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:14.338+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:14.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:14.338+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:14.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:14.536+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:14.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:14 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:14.536+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:15.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:15.289+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:15.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:15.289+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:15.623 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:15.441+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:15.623 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:15.441+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:47.154433+0000 front 2023-10-03T07:14:47.154391+0000 (oldest deadline 2023-10-03T07:15:11.854478+0000) 2023-10-03T07:15:15.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:15.498+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:15.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:15 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:15.498+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:16.545 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:16.448+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.153313+0000 front 2023-10-03T07:14:37.153340+0000 (oldest deadline 2023-10-03T07:15:02.453517+0000) 2023-10-03T07:15:16.545 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5[114022]: 2023-10-03T07:15:16.448+0000 7f40c6d10700 -1 osd.5 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:47.154433+0000 front 2023-10-03T07:14:47.154391+0000 (oldest deadline 2023-10-03T07:15:11.854478+0000) 2023-10-03T07:15:16.546 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:16.258+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:16.546 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:16.258+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:16.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:16.545+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:16.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:16 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:16.545+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:17.565 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:17 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:17.252+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:17.565 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:17 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:17.252+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:17.565 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:17 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:17.499+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:17.566 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:17 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:17.499+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:17.871 INFO:journalctl@ceph.osd.5.smithi139.stdout:Oct 03 07:15:17 smithi139 bash[125255]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-5 2023-10-03T07:15:18.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:18.262+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:18.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:18.262+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:18.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:18.480+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:18.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:18.480+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:18.649 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.5.service' 2023-10-03T07:15:18.688 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:15:18.689 INFO:tasks.cephadm.osd.5:Stopped osd.5 2023-10-03T07:15:18.689 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2023-10-03T07:15:18.689 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.6 2023-10-03T07:15:18.943 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 systemd[1]: Stopping Ceph osd.6 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:15:19.263 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:18.943+0000 7fce0b6bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:15:19.263 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:18.943+0000 7fce0b6bb700 -1 osd.6 55 *** Got signal Terminated *** 2023-10-03T07:15:19.263 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:18 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:18.943+0000 7fce0b6bb700 -1 osd.6 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:15:19.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:19 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:19.263+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:19.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:19 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:19.263+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:19.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:19 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:19.263+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:50.251423+0000 front 2023-10-03T07:14:50.251309+0000 (oldest deadline 2023-10-03T07:15:19.051744+0000) 2023-10-03T07:15:19.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:19 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:19.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:19.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:19 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:19.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:20.536 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:20 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:20.268+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:20.537 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:20 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:20.268+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:20.537 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:20 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:20.268+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:50.251423+0000 front 2023-10-03T07:14:50.251309+0000 (oldest deadline 2023-10-03T07:15:19.051744+0000) 2023-10-03T07:15:20.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:20 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:20.536+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:20.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:20 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:20.536+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:21.533 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:21 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:21.266+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:21.533 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:21 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:21.266+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:21.533 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:21 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:21.266+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:50.251423+0000 front 2023-10-03T07:14:50.251309+0000 (oldest deadline 2023-10-03T07:15:19.051744+0000) 2023-10-03T07:15:21.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:21 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:21.533+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:21.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:21 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:21.533+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:22.505 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:22.237+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:22.505 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:22.237+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:22.505 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:22.237+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:50.251423+0000 front 2023-10-03T07:14:50.251309+0000 (oldest deadline 2023-10-03T07:15:19.051744+0000) 2023-10-03T07:15:22.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:22.505+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:22.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:22.505+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:22.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:22 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:22.505+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:23.621 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:23.267+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:39.149703+0000 front 2023-10-03T07:14:39.149750+0000 (oldest deadline 2023-10-03T07:15:03.849841+0000) 2023-10-03T07:15:23.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:23.267+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:46.750649+0000 front 2023-10-03T07:14:46.750685+0000 (oldest deadline 2023-10-03T07:15:10.250738+0000) 2023-10-03T07:15:23.622 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6[118283]: 2023-10-03T07:15:23.267+0000 7fce074b0700 -1 osd.6 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:50.251423+0000 front 2023-10-03T07:14:50.251309+0000 (oldest deadline 2023-10-03T07:15:19.051744+0000) 2023-10-03T07:15:23.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:23.485+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:23.622 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:23.485+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:23.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:23.485+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:23.623 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:23 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:23.485+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:24.866 INFO:journalctl@ceph.osd.6.smithi139.stdout:Oct 03 07:15:24 smithi139 bash[125483]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-6 2023-10-03T07:15:24.866 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:24 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:24.528+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:24.867 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:24 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:24.528+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:24.867 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:24 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:24.528+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:24.867 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:24 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:24.528+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:25.549 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.6.service' 2023-10-03T07:15:25.590 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:15:25.590 INFO:tasks.cephadm.osd.6:Stopped osd.6 2023-10-03T07:15:25.590 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2023-10-03T07:15:25.591 DEBUG:teuthology.orchestra.run.smithi139:> sudo systemctl stop ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.7 2023-10-03T07:15:25.846 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:25.847 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:25.847 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:25.847 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.512+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:25.847 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 systemd[1]: Stopping Ceph osd.7 for 8ad20b58-61bb-11ee-8db5-212e2dc638e7... 2023-10-03T07:15:26.121 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.846+0000 7f7156109700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2023-10-03T07:15:26.121 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.846+0000 7f7156109700 -1 osd.7 55 *** Got signal Terminated *** 2023-10-03T07:15:26.122 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:25 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:25.846+0000 7f7156109700 -1 osd.7 55 *** Immediate shutdown (osd_fast_shutdown=true) *** 2023-10-03T07:15:26.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:26 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:26.541+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:26.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:26 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:26.541+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:26.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:26 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:26.541+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:26.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:26 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:26.541+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:27.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:27 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:27.576+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:27.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:27 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:27.576+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:27.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:27 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:27.576+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:27.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:27 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:27.576+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:28.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:28 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:28.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:28.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:28 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:28.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:28.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:28 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:28.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:28.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:28 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:28.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:29.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:29 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:29.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:29.871 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:29 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:29.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:29.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:29 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:29.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:29.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:29 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:29.568+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:30.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:30 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:30.519+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6806 osd.0 since back 2023-10-03T07:14:37.795245+0000 front 2023-10-03T07:14:37.795359+0000 (oldest deadline 2023-10-03T07:15:02.495358+0000) 2023-10-03T07:15:30.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:30 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:30.519+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.31:6814 osd.1 since back 2023-10-03T07:14:48.797126+0000 front 2023-10-03T07:14:48.797155+0000 (oldest deadline 2023-10-03T07:15:14.097333+0000) 2023-10-03T07:15:30.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:30 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:30.519+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6804 osd.2 since back 2023-10-03T07:14:55.798351+0000 front 2023-10-03T07:14:55.798397+0000 (oldest deadline 2023-10-03T07:15:21.698282+0000) 2023-10-03T07:15:30.872 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:30 smithi139 ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7[122440]: 2023-10-03T07:15:30.519+0000 7f71516fd700 -1 osd.7 55 heartbeat_check: no reply from 172.21.15.114:6812 osd.3 since back 2023-10-03T07:15:01.698848+0000 front 2023-10-03T07:15:01.698810+0000 (oldest deadline 2023-10-03T07:15:23.398725+0000) 2023-10-03T07:15:31.621 INFO:journalctl@ceph.osd.7.smithi139.stdout:Oct 03 07:15:31 smithi139 bash[125718]: ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7-osd-7 2023-10-03T07:15:32.471 DEBUG:teuthology.orchestra.run.smithi139:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8ad20b58-61bb-11ee-8db5-212e2dc638e7@osd.7.service' 2023-10-03T07:15:32.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-10-03T07:15:32.509 INFO:tasks.cephadm.osd.7:Stopped osd.7 2023-10-03T07:15:32.509 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force --keep-logs 2023-10-03T07:15:32.695 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:34.344 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force --keep-logs 2023-10-03T07:15:34.530 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:35.268 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force --keep-logs 2023-10-03T07:15:35.455 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:36.187 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:15:36.228 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:15:36.262 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2023-10-03T07:15:36.292 INFO:tasks.cephadm:Archiving crash dumps... 2023-10-03T07:15:36.293 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi031/crash 2023-10-03T07:15:36.294 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash -- . 2023-10-03T07:15:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:tar: /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-03T07:15:36.342 INFO:teuthology.orchestra.run.smithi031.stderr:tar: Error is not recoverable: exiting now 2023-10-03T07:15:36.344 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi114/crash 2023-10-03T07:15:36.345 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash -- . 2023-10-03T07:15:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:tar: /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-03T07:15:36.415 INFO:teuthology.orchestra.run.smithi114.stderr:tar: Error is not recoverable: exiting now 2023-10-03T07:15:36.417 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi139/crash 2023-10-03T07:15:36.418 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash -- . 2023-10-03T07:15:36.452 INFO:teuthology.orchestra.run.smithi139.stderr:tar: /var/lib/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/crash: Cannot open: No such file or directory 2023-10-03T07:15:36.452 INFO:teuthology.orchestra.run.smithi139.stderr:tar: Error is not recoverable: exiting now 2023-10-03T07:15:36.453 INFO:tasks.cephadm:Checking cluster log for badness... 2023-10-03T07:15:36.453 DEBUG:teuthology.orchestra.run.smithi031:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | head -n 1 2023-10-03T07:15:36.489 INFO:teuthology.orchestra.run.smithi031.stderr:grep: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph.log: No such file or directory 2023-10-03T07:15:36.491 INFO:tasks.cephadm:Compressing logs... 2023-10-03T07:15:36.491 DEBUG:teuthology.orchestra.run.smithi031:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T07:15:36.534 DEBUG:teuthology.orchestra.run.smithi114:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T07:15:36.536 DEBUG:teuthology.orchestra.run.smithi139:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-10-03T07:15:36.563 INFO:teuthology.orchestra.run.smithi139.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- : No such file or directory/var/log/ceph/cephadm.log 2023-10-03T07:15:36.564 INFO:teuthology.orchestra.run.smithi139.stderr: 2023-10-03T07:15:36.564 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.6.log 2023-10-03T07:15:36.564 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log 2023-10-03T07:15:36.564 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.7.log 2023-10-03T07:15:36.566 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log: 90.8% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-03T07:15:36.566 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.c.log 2023-10-03T07:15:36.566 INFO:teuthology.orchestra.run.smithi139.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.5.log 2023-10-03T07:15:36.567 INFO:teuthology.orchestra.run.smithi114.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-10-03T07:15:36.567 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-10-03T07:15:36.567 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.2.log 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.3.log 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.2.log: 89.8% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.b.log 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.4.log 2023-10-03T07:15:36.568 INFO:teuthology.orchestra.run.smithi114.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.3.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.b.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.4.log: 91.7% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log.gz 2023-10-03T07:15:36.569 INFO:teuthology.orchestra.run.smithi139.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.7.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.c.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.5.log: 91.7% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log.gz 2023-10-03T07:15:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-10-03T07:15:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:: No such file or directory 2023-10-03T07:15:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log 2023-10-03T07:15:36.570 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.0.log 2023-10-03T07:15:36.571 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mgr.x.log 2023-10-03T07:15:36.571 INFO:teuthology.orchestra.run.smithi031.stderr:gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.a.log 2023-10-03T07:15:36.571 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.0.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.1.log 2023-10-03T07:15:36.571 INFO:teuthology.orchestra.run.smithi031.stderr:/var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.a.log: /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.1.log: 91.5% -- replaced with /var/log/ceph/cephadm.log.gz 2023-10-03T07:15:36.572 INFO:teuthology.orchestra.run.smithi031.stderr: 91.7% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-volume.log.gz 2023-10-03T07:15:36.613 INFO:teuthology.orchestra.run.smithi114.stderr: 92.7% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.4.log.gz 2023-10-03T07:15:36.619 INFO:teuthology.orchestra.run.smithi139.stderr: 92.8% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.5.log.gz 2023-10-03T07:15:36.621 INFO:teuthology.orchestra.run.smithi139.stderr: 92.9% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.6.log.gz 2023-10-03T07:15:36.624 INFO:teuthology.orchestra.run.smithi139.stderr: 92.5% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.c.log.gz 2023-10-03T07:15:36.625 INFO:teuthology.orchestra.run.smithi114.stderr: 92.4% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.b.log.gz 2023-10-03T07:15:36.630 INFO:teuthology.orchestra.run.smithi139.stderr: 93.3% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.7.log.gz 2023-10-03T07:15:36.630 INFO:teuthology.orchestra.run.smithi031.stderr: 92.9% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.0.log.gz 2023-10-03T07:15:36.631 INFO:teuthology.orchestra.run.smithi139.stderr: 2023-10-03T07:15:36.631 INFO:teuthology.orchestra.run.smithi139.stderr:real 0m0.079s 2023-10-03T07:15:36.632 INFO:teuthology.orchestra.run.smithi139.stderr:user 0m0.241s 2023-10-03T07:15:36.632 INFO:teuthology.orchestra.run.smithi139.stderr:sys 0m0.023s 2023-10-03T07:15:36.641 INFO:teuthology.orchestra.run.smithi114.stderr: 92.9% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.3.log.gz 2023-10-03T07:15:36.651 INFO:teuthology.orchestra.run.smithi031.stderr: 89.6% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mgr.x.log.gz 2023-10-03T07:15:36.657 INFO:teuthology.orchestra.run.smithi031.stderr: 93.1% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.1.log.gz 2023-10-03T07:15:36.661 INFO:teuthology.orchestra.run.smithi114.stderr: 93.4% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-osd.2.log.gz 2023-10-03T07:15:36.662 INFO:teuthology.orchestra.run.smithi114.stderr: 2023-10-03T07:15:36.662 INFO:teuthology.orchestra.run.smithi114.stderr:real 0m0.110s 2023-10-03T07:15:36.662 INFO:teuthology.orchestra.run.smithi114.stderr:user 0m0.283s 2023-10-03T07:15:36.663 INFO:teuthology.orchestra.run.smithi114.stderr:sys 0m0.024s 2023-10-03T07:15:36.787 INFO:teuthology.orchestra.run.smithi031.stderr: 91.1% -- replaced with /var/log/ceph/8ad20b58-61bb-11ee-8db5-212e2dc638e7/ceph-mon.a.log.gz 2023-10-03T07:15:36.789 INFO:teuthology.orchestra.run.smithi031.stderr: 2023-10-03T07:15:36.789 INFO:teuthology.orchestra.run.smithi031.stderr:real 0m0.236s 2023-10-03T07:15:36.789 INFO:teuthology.orchestra.run.smithi031.stderr:user 0m0.452s 2023-10-03T07:15:36.789 INFO:teuthology.orchestra.run.smithi031.stderr:sys 0m0.036s 2023-10-03T07:15:36.790 INFO:tasks.cephadm:Archiving logs... 2023-10-03T07:15:36.790 DEBUG:teuthology.misc:Transferring archived files from smithi031:/var/log/ceph to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi031/log 2023-10-03T07:15:36.791 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T07:15:36.930 DEBUG:teuthology.misc:Transferring archived files from smithi114:/var/log/ceph to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi114/log 2023-10-03T07:15:36.931 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T07:15:37.003 DEBUG:teuthology.misc:Transferring archived files from smithi139:/var/log/ceph to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi139/log 2023-10-03T07:15:37.003 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /var/log/ceph -- . 2023-10-03T07:15:37.067 INFO:tasks.cephadm:Removing cluster... 2023-10-03T07:15:37.068 DEBUG:teuthology.orchestra.run.smithi031:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force 2023-10-03T07:15:37.272 INFO:teuthology.orchestra.run.smithi031.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:37.436 DEBUG:teuthology.orchestra.run.smithi114:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force 2023-10-03T07:15:37.617 INFO:teuthology.orchestra.run.smithi114.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:37.759 DEBUG:teuthology.orchestra.run.smithi139:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8ad20b58-61bb-11ee-8db5-212e2dc638e7 --force 2023-10-03T07:15:37.942 INFO:teuthology.orchestra.run.smithi139.stdout:Deleting cluster with fsid: 8ad20b58-61bb-11ee-8db5-212e2dc638e7 2023-10-03T07:15:38.083 INFO:tasks.cephadm:Removing cephadm ... 2023-10-03T07:15:38.084 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-03T07:15:38.107 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-03T07:15:38.125 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf /home/ubuntu/cephtest/cephadm 2023-10-03T07:15:38.144 INFO:tasks.cephadm:Teardown complete 2023-10-03T07:15:38.144 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-10-03T07:15:38.159 INFO:teuthology.task.clock:Checking final clock skew... 2023-10-03T07:15:38.160 DEBUG:teuthology.orchestra.run.smithi031:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:15:38.162 DEBUG:teuthology.orchestra.run.smithi114:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:15:38.168 DEBUG:teuthology.orchestra.run.smithi139:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-10-03T07:15:38.181 INFO:teuthology.orchestra.run.smithi031.stderr:bash: ntpq: command not found 2023-10-03T07:15:38.182 INFO:teuthology.orchestra.run.smithi114.stderr:bash: ntpq: command not found 2023-10-03T07:15:38.186 INFO:teuthology.orchestra.run.smithi031.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:15:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:=============================================================================== 2023-10-03T07:15:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 -100us[ -100us] +/- 75ms 2023-10-03T07:15:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:^* hv02.front.sepia.ceph.com 3 6 377 34 -501us[ -531us] +/- 51ms 2023-10-03T07:15:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +860us[ +830us] +/- 80ms 2023-10-03T07:15:38.187 INFO:teuthology.orchestra.run.smithi031.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:=============================================================================== 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 35 -97us[ -112us] +/- 75ms 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:^* hv02.front.sepia.ceph.com 3 6 377 35 -501us[ -516us] +/- 51ms 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +866us[ +851us] +/- 80ms 2023-10-03T07:15:38.188 INFO:teuthology.orchestra.run.smithi114.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:15:38.201 INFO:teuthology.orchestra.run.smithi139.stderr:bash: ntpq: command not found 2023-10-03T07:15:38.205 INFO:teuthology.orchestra.run.smithi139.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-10-03T07:15:38.205 INFO:teuthology.orchestra.run.smithi139.stdout:=============================================================================== 2023-10-03T07:15:38.205 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 34 -86us[ -113us] +/- 75ms 2023-10-03T07:15:38.206 INFO:teuthology.orchestra.run.smithi139.stdout:^* hv02.front.sepia.ceph.com 3 6 377 33 -498us[ -524us] +/- 51ms 2023-10-03T07:15:38.206 INFO:teuthology.orchestra.run.smithi139.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 36 +866us[ +839us] +/- 80ms 2023-10-03T07:15:38.206 INFO:teuthology.orchestra.run.smithi139.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-10-03T07:15:38.206 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-10-03T07:15:38.214 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-10-03T07:15:38.215 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-10-03T07:15:38.264 DEBUG:teuthology.orchestra.run.smithi031:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-03T07:15:38.372 DEBUG:teuthology.orchestra.run.smithi114:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-03T07:15:38.461 DEBUG:teuthology.orchestra.run.smithi139:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-10-03T07:15:38.551 DEBUG:teuthology.orchestra.run.smithi031:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T07:15:38.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:38.592 DEBUG:teuthology.orchestra.run.smithi114:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T07:15:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:38.625 DEBUG:teuthology.orchestra.run.smithi139:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-10-03T07:15:38.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:38.657 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-10-03T07:15:38.667 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-10-03T07:15:38.714 INFO:teuthology.task.internal:Duration was 964.057172 seconds 2023-10-03T07:15:38.715 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-10-03T07:15:38.764 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-10-03T07:15:38.764 DEBUG:teuthology.orchestra.run.smithi031:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T07:15:38.767 DEBUG:teuthology.orchestra.run.smithi114:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T07:15:38.769 DEBUG:teuthology.orchestra.run.smithi139:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-10-03T07:15:38.814 INFO:teuthology.orchestra.run.smithi114.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T07:15:38.815 INFO:teuthology.orchestra.run.smithi139.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T07:15:38.824 INFO:teuthology.orchestra.run.smithi031.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-10-03T07:15:39.256 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-10-03T07:15:39.256 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi031.front.sepia.ceph.com 2023-10-03T07:15:39.256 DEBUG:teuthology.orchestra.run.smithi031:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T07:15:39.286 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi114.front.sepia.ceph.com 2023-10-03T07:15:39.286 DEBUG:teuthology.orchestra.run.smithi114:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T07:15:39.309 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi139.front.sepia.ceph.com 2023-10-03T07:15:39.310 DEBUG:teuthology.orchestra.run.smithi139:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-10-03T07:15:39.333 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-10-03T07:15:39.334 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T07:15:39.336 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T07:15:39.352 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-10-03T07:15:39.416 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-10-03T07:15:39.416 DEBUG:teuthology.orchestra.run.smithi031:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T07:15:39.419 DEBUG:teuthology.orchestra.run.smithi114:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T07:15:39.436 DEBUG:teuthology.orchestra.run.smithi139:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-10-03T07:15:39.532 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-10-03T07:15:39.542 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-10-03T07:15:39.542 DEBUG:teuthology.orchestra.run.smithi031:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T07:15:39.575 DEBUG:teuthology.orchestra.run.smithi114:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T07:15:39.603 DEBUG:teuthology.orchestra.run.smithi139:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-10-03T07:15:39.630 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-10-03T07:15:39.640 DEBUG:teuthology.orchestra.run.smithi031:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.642 DEBUG:teuthology.orchestra.run.smithi114:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.645 DEBUG:teuthology.orchestra.run.smithi139:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.669 INFO:teuthology.orchestra.run.smithi114.stdout:kernel.core_pattern = core 2023-10-03T07:15:39.674 INFO:teuthology.orchestra.run.smithi031.stdout:kernel.core_pattern = core 2023-10-03T07:15:39.698 INFO:teuthology.orchestra.run.smithi139.stdout:kernel.core_pattern = core 2023-10-03T07:15:39.743 DEBUG:teuthology.orchestra.run.smithi031:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:39.768 DEBUG:teuthology.orchestra.run.smithi114:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:39.786 DEBUG:teuthology.orchestra.run.smithi139:> test -e /home/ubuntu/cephtest/archive/coredump 2023-10-03T07:15:39.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:39.804 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-10-03T07:15:39.820 INFO:teuthology.task.internal:Transferring archived files... 2023-10-03T07:15:39.821 DEBUG:teuthology.misc:Transferring archived files from smithi031:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi031 2023-10-03T07:15:39.821 DEBUG:teuthology.orchestra.run.smithi031:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T07:15:39.872 DEBUG:teuthology.misc:Transferring archived files from smithi114:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi114 2023-10-03T07:15:39.873 DEBUG:teuthology.orchestra.run.smithi114:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T07:15:39.913 DEBUG:teuthology.misc:Transferring archived files from smithi139:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514/remote/smithi139 2023-10-03T07:15:39.913 DEBUG:teuthology.orchestra.run.smithi139:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-10-03T07:15:39.953 INFO:teuthology.task.internal:Removing archive directory... 2023-10-03T07:15:39.954 DEBUG:teuthology.orchestra.run.smithi031:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T07:15:39.956 DEBUG:teuthology.orchestra.run.smithi114:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T07:15:39.959 DEBUG:teuthology.orchestra.run.smithi139:> rm -rf -- /home/ubuntu/cephtest/archive 2023-10-03T07:15:40.004 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-10-03T07:15:40.017 INFO:teuthology.task.internal:Not uploading archives. 2023-10-03T07:15:40.017 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-10-03T07:15:40.041 INFO:teuthology.task.internal:Tidying up after the test... 2023-10-03T07:15:40.042 DEBUG:teuthology.orchestra.run.smithi031:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T07:15:40.044 DEBUG:teuthology.orchestra.run.smithi114:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T07:15:40.046 DEBUG:teuthology.orchestra.run.smithi139:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-10-03T07:15:40.061 INFO:teuthology.orchestra.run.smithi114.stdout: 525982 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 3 07:15 /home/ubuntu/cephtest 2023-10-03T07:15:40.064 INFO:teuthology.orchestra.run.smithi139.stdout: 526080 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Oct 3 07:15 /home/ubuntu/cephtest 2023-10-03T07:15:40.064 INFO:teuthology.orchestra.run.smithi031.stdout: 525357 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Oct 3 07:15 /home/ubuntu/cephtest 2023-10-03T07:15:40.065 INFO:teuthology.orchestra.run.smithi031.stdout: 527033 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 07:13 /home/ubuntu/cephtest/mnt.0 2023-10-03T07:15:40.065 INFO:teuthology.orchestra.run.smithi031.stdout: 527034 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Oct 3 07:14 /home/ubuntu/cephtest/mnt.0/client.0 2023-10-03T07:15:40.065 INFO:teuthology.orchestra.run.smithi031.stdout: 527113 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Oct 3 07:14 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2023-10-03T07:15:40.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-10-03T07:15:40.066 INFO:teuthology.orchestra.run.smithi031.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2023-10-03T07:15:40.067 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_54e62bcbac4e53d9685e08328b790d3b20d71cae/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi031 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T07:15:40.072 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-10-03T07:15:40.091 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-10-03T07:15:40.142 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on smithi031 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2023-10-03T07:15:40.142 INFO:teuthology.nuke:Checking targets against current locks 2023-10-03T07:15:40.171 DEBUG:teuthology.nuke:shortname: smithi031 2023-10-03T07:15:40.171 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T07:15:40.197 DEBUG:teuthology.nuke:shortname: smithi114 2023-10-03T07:15:40.197 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T07:15:40.202 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi031.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.388335', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGJc0g7r8WVMYr3AaaywfLJYIjVH44RBAwx116cDxldOxZBLOdm1Ru8mwFiIjNFf2ZFRQUuRhpQm71jFXkvcdW8='} 2023-10-03T07:15:40.224 DEBUG:teuthology.nuke:shortname: smithi139 2023-10-03T07:15:40.224 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-10-03T07:15:40.229 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi114.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.386631', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCaaEs8/rx7U0jxdpTvEWhxgBfNhU/ko2AJyjCIlqYMQmHaSLe90uVGdhnF9cOuGFEPmn0xb0a+qqw9Xwya9rHw='} 2023-10-03T07:15:40.253 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi139.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/bdavidov-2023-10-03_06:50:26-rbd:nvmeof-wip-barakda-cephadm-nvmeof-distro-default-smithi/7409514', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-10-03 06:51:22.389811', 'locked_by': 'scheduled_bdavidov@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKidOfiR9a47mR8/jcWpflnygMci/DvOOBqb3vVe0bHirAMu4JiWggY/PYr/YCkwyZcYSR8uLqzCstac9NbhIaY='} 2023-10-03T07:15:40.259 INFO:teuthology.orchestra.console.smithi031:Power off 2023-10-03T07:15:40.259 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T07:15:40.284 DEBUG:teuthology.orchestra.console.smithi031:power off output: Chassis Power Control: Down/Off 2023-10-03T07:15:40.284 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:40.300 INFO:teuthology.orchestra.console.smithi114:Power off 2023-10-03T07:15:40.300 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T07:15:40.309 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-03T07:15:40.321 DEBUG:teuthology.orchestra.console.smithi114:power off output: Chassis Power Control: Down/Off 2023-10-03T07:15:40.321 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:40.335 INFO:teuthology.orchestra.console.smithi139:Power off 2023-10-03T07:15:40.335 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-10-03T07:15:40.343 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-10-03T07:15:40.356 DEBUG:teuthology.orchestra.console.smithi139:power off output: Chassis Power Control: Down/Off 2023-10-03T07:15:40.356 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:40.379 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2023-10-03T07:15:44.311 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:44.344 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:44.380 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:44.426 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-03T07:15:44.454 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-10-03T07:15:44.491 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is on 2023-10-03T07:15:48.431 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:48.455 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:48.492 DEBUG:teuthology.orchestra.console.smithi139:pexpect command: ipmitool -H smithi139.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:48.544 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is on 2023-10-03T07:15:48.567 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is on 2023-10-03T07:15:48.602 DEBUG:teuthology.orchestra.console.smithi139:check power output: Chassis Power is off 2023-10-03T07:15:48.703 INFO:teuthology.orchestra.console.smithi139:Power off completed 2023-10-03T07:15:52.549 DEBUG:teuthology.orchestra.console.smithi031:pexpect command: ipmitool -H smithi031.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:52.568 DEBUG:teuthology.orchestra.console.smithi114:pexpect command: ipmitool -H smithi114.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-10-03T07:15:52.664 DEBUG:teuthology.orchestra.console.smithi031:check power output: Chassis Power is off 2023-10-03T07:15:52.678 DEBUG:teuthology.orchestra.console.smithi114:check power output: Chassis Power is off 2023-10-03T07:15:52.765 INFO:teuthology.orchestra.console.smithi031:Power off completed 2023-10-03T07:15:52.779 INFO:teuthology.orchestra.console.smithi114:Power off completed 2023-10-03T07:15:52.885 INFO:teuthology.run:Summary data: description: rbd:nvmeof/{0-single-container-host base/install cluster/{fixed-3 openstack} workloads/cephadm_nvmeof} duration: 964.0571715831757 failure_reason: 'Command failed (workunit test cephadm/test_nvme_etc_hosts.sh) on smithi031 with status 127: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=b6b4a26de3cb4337d4479f11f356af936649f829 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/cephadm/test_nvme_etc_hosts.sh''' owner: scheduled_bdavidov@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=4637935d25af414d8cc9d6d3c20414d9 status: fail success: false 2023-10-03T07:15:52.885 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-10-03T07:15:52.974 INFO:teuthology.run:FAIL